2024-08-18T21:08:21.388 INFO:root:teuthology version: 1.2.0 2024-08-18T21:08:21.401 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-18T21:08:21.483 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859229 branch: main description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: ceph-qa@ceph.com first_in_suite: false job_id: '7859229' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b workunit: branch: main sha1: 964adda23da9f7b057cb34fdc0b82aeb7ed65324 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1858 sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b sleep_before_teardown: 0 subset: 29813/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 964adda23da9f7b057cb34fdc0b82aeb7ed65324 targets: smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGlgS/KRyUVvJor4P9++ch3LVvvTLm/mgTqIBoGF18AZMoCXDxKNcmGmTxG0OhmkN91uWRP9yqwhe7ezt3oSqVw= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 67db276914d29505c9707d04f3a3b8fb9d8aee4b timestamp: 2024-08-18_20:00:17 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-08-18T21:08:21.483 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa; will attempt to use it 2024-08-18T21:08:21.484 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks 2024-08-18T21:08:21.484 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-08-18T21:08:21.486 INFO:teuthology.task.internal:Checking packages... 2024-08-18T21:08:21.510 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '797669399b32d2915e64eb6d4b1257ae3c50f80b' 2024-08-18T21:08:21.510 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-18T21:08:21.510 INFO:teuthology.packaging:ref: None 2024-08-18T21:08:21.510 INFO:teuthology.packaging:tag: None 2024-08-18T21:08:21.511 INFO:teuthology.packaging:branch: main 2024-08-18T21:08:21.511 INFO:teuthology.packaging:sha1: 797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-18T21:08:21.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-08-18T21:08:21.775 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-5847-g79766939-1jammy 2024-08-18T21:08:21.777 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-08-18T21:08:21.784 INFO:teuthology.task.internal:no buildpackages task found 2024-08-18T21:08:21.784 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-18T21:08:21.791 INFO:teuthology.task.internal:Saving configuration 2024-08-18T21:08:21.800 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-18T21:08:21.807 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-18T21:08:21.831 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859229', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-18 21:02:11.732040', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIODio4IBkY5ziJjxobSJnYtePQeziFkaetFvCRg2jpcB'} 2024-08-18T21:08:21.831 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-18T21:08:21.838 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-08-18T21:08:21.838 INFO:teuthology.run_tasks:Running task console_log... 2024-08-18T21:08:21.871 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbb15a73760>, signals=[15]) 2024-08-18T21:08:21.872 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-18T21:08:21.878 INFO:teuthology.task.internal:Opening connections... 2024-08-18T21:08:21.878 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2024-08-18T21:08:21.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T21:08:21.967 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-18T21:08:21.975 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2024-08-18T21:08:22.083 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2024-08-18T21:08:22.083 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2024-08-18T21:08:22.129 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-08-18T21:08:22.129 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="Ubuntu" 2024-08-18T21:08:22.130 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="22.04" 2024-08-18T21:08:22.130 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-08-18T21:08:22.130 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_CODENAME=jammy 2024-08-18T21:08:22.130 INFO:teuthology.orchestra.run.smithi099.stdout:ID=ubuntu 2024-08-18T21:08:22.130 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE=debian 2024-08-18T21:08:22.130 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://www.ubuntu.com/" 2024-08-18T21:08:22.130 INFO:teuthology.orchestra.run.smithi099.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-08-18T21:08:22.130 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-08-18T21:08:22.130 INFO:teuthology.orchestra.run.smithi099.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-08-18T21:08:22.130 INFO:teuthology.orchestra.run.smithi099.stdout:UBUNTU_CODENAME=jammy 2024-08-18T21:08:22.131 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2024-08-18T21:08:22.154 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-18T21:08:22.162 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-18T21:08:22.168 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-18T21:08:22.168 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2024-08-18T21:08:22.177 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-18T21:08:22.184 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-18T21:08:22.184 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2024-08-18T21:08:22.227 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-18T21:08:22.259 INFO:teuthology.run_tasks:Running task kernel... 2024-08-18T21:08:22.274 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-18T21:08:22.274 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-18T21:08:22.274 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-18T21:08:22.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T21:08:22.279 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-08-18T21:08:22.326 INFO:teuthology.orchestra.run.smithi099.stdout:5.15.0-56-generic 2024-08-18T21:08:22.326 INFO:teuthology.task.kernel:Running kernel on smithi099: 5.15.0-56-generic 2024-08-18T21:08:22.326 DEBUG:teuthology.orchestra.run.smithi099:> sudo apt-get clean 2024-08-18T21:08:22.555 DEBUG:teuthology.orchestra.run.smithi099:> sudo apt-get update 2024-08-18T21:08:22.719 INFO:teuthology.orchestra.run.smithi099.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-18T21:08:22.734 INFO:teuthology.orchestra.run.smithi099.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-08-18T21:08:22.800 INFO:teuthology.orchestra.run.smithi099.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-08-18T21:08:22.834 INFO:teuthology.orchestra.run.smithi099.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-08-18T21:08:22.943 INFO:teuthology.orchestra.run.smithi099.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [684 kB] 2024-08-18T21:08:23.014 INFO:teuthology.orchestra.run.smithi099.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,948 kB] 2024-08-18T21:08:23.049 INFO:teuthology.orchestra.run.smithi099.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [345 kB] 2024-08-18T21:08:23.052 INFO:teuthology.orchestra.run.smithi099.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-08-18T21:08:23.052 INFO:teuthology.orchestra.run.smithi099.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,322 kB] 2024-08-18T21:08:23.072 INFO:teuthology.orchestra.run.smithi099.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.9 kB] 2024-08-18T21:08:23.073 INFO:teuthology.orchestra.run.smithi099.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [401 kB] 2024-08-18T21:08:23.077 INFO:teuthology.orchestra.run.smithi099.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [604 B] 2024-08-18T21:08:23.077 INFO:teuthology.orchestra.run.smithi099.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,111 kB] 2024-08-18T21:08:23.086 INFO:teuthology.orchestra.run.smithi099.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [725 kB] 2024-08-18T21:08:23.092 INFO:teuthology.orchestra.run.smithi099.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [259 kB] 2024-08-18T21:08:23.094 INFO:teuthology.orchestra.run.smithi099.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [25.9 kB] 2024-08-18T21:08:23.094 INFO:teuthology.orchestra.run.smithi099.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-08-18T21:08:23.117 INFO:teuthology.orchestra.run.smithi099.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-08-18T21:08:23.117 INFO:teuthology.orchestra.run.smithi099.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-08-18T21:08:23.117 INFO:teuthology.orchestra.run.smithi099.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-08-18T21:08:23.117 INFO:teuthology.orchestra.run.smithi099.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-08-18T21:08:23.118 INFO:teuthology.orchestra.run.smithi099.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-08-18T21:08:23.119 INFO:teuthology.orchestra.run.smithi099.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-08-18T21:08:23.119 INFO:teuthology.orchestra.run.smithi099.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-08-18T21:08:23.119 INFO:teuthology.orchestra.run.smithi099.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.2 kB] 2024-08-18T21:08:23.136 INFO:teuthology.orchestra.run.smithi099.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-08-18T21:08:23.136 INFO:teuthology.orchestra.run.smithi099.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-08-18T21:08:23.136 INFO:teuthology.orchestra.run.smithi099.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-08-18T21:08:23.272 INFO:teuthology.orchestra.run.smithi099.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,729 kB] 2024-08-18T21:08:23.687 INFO:teuthology.orchestra.run.smithi099.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [524 kB] 2024-08-18T21:08:23.703 INFO:teuthology.orchestra.run.smithi099.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [287 kB] 2024-08-18T21:08:23.712 INFO:teuthology.orchestra.run.smithi099.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.2 kB] 2024-08-18T21:08:23.712 INFO:teuthology.orchestra.run.smithi099.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.3 kB] 2024-08-18T21:08:23.713 INFO:teuthology.orchestra.run.smithi099.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,247 kB] 2024-08-18T21:08:23.809 INFO:teuthology.orchestra.run.smithi099.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [388 kB] 2024-08-18T21:08:23.820 INFO:teuthology.orchestra.run.smithi099.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [572 B] 2024-08-18T21:08:23.820 INFO:teuthology.orchestra.run.smithi099.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [890 kB] 2024-08-18T21:08:23.872 INFO:teuthology.orchestra.run.smithi099.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [621 kB] 2024-08-18T21:08:23.886 INFO:teuthology.orchestra.run.smithi099.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [175 kB] 2024-08-18T21:08:23.889 INFO:teuthology.orchestra.run.smithi099.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.0 kB] 2024-08-18T21:08:23.890 INFO:teuthology.orchestra.run.smithi099.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-08-18T21:08:23.891 INFO:teuthology.orchestra.run.smithi099.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-08-18T21:08:23.891 INFO:teuthology.orchestra.run.smithi099.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-08-18T21:08:23.950 INFO:teuthology.orchestra.run.smithi099.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-08-18T21:08:29.194 INFO:teuthology.orchestra.run.smithi099.stdout:Fetched 15.5 MB in 2s (7,530 kB/s) 2024-08-18T21:08:30.577 INFO:teuthology.orchestra.run.smithi099.stdout:Reading package lists... 2024-08-18T21:08:30.595 DEBUG:teuthology.orchestra.run.smithi099:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-08-18T21:08:30.658 INFO:teuthology.orchestra.run.smithi099.stdout:Reading package lists... 2024-08-18T21:08:30.866 INFO:teuthology.orchestra.run.smithi099.stdout:Building dependency tree... 2024-08-18T21:08:30.866 INFO:teuthology.orchestra.run.smithi099.stdout:Reading state information... 2024-08-18T21:08:31.040 INFO:teuthology.orchestra.run.smithi099.stdout:The following packages were automatically installed and are no longer required: 2024-08-18T21:08:31.040 INFO:teuthology.orchestra.run.smithi099.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-18T21:08:31.040 INFO:teuthology.orchestra.run.smithi099.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-18T21:08:31.041 INFO:teuthology.orchestra.run.smithi099.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-18T21:08:31.041 INFO:teuthology.orchestra.run.smithi099.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-08-18T21:08:31.041 INFO:teuthology.orchestra.run.smithi099.stdout: python2.7-minimal 2024-08-18T21:08:31.041 INFO:teuthology.orchestra.run.smithi099.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-18T21:08:31.042 INFO:teuthology.orchestra.run.smithi099.stdout:The following additional packages will be installed: 2024-08-18T21:08:31.042 INFO:teuthology.orchestra.run.smithi099.stdout: linux-generic linux-headers-5.15.0-118 linux-headers-5.15.0-118-generic 2024-08-18T21:08:31.042 INFO:teuthology.orchestra.run.smithi099.stdout: linux-headers-generic linux-image-5.15.0-118-generic 2024-08-18T21:08:31.042 INFO:teuthology.orchestra.run.smithi099.stdout: linux-modules-5.15.0-118-generic linux-modules-extra-5.15.0-118-generic 2024-08-18T21:08:31.043 INFO:teuthology.orchestra.run.smithi099.stdout:Suggested packages: 2024-08-18T21:08:31.044 INFO:teuthology.orchestra.run.smithi099.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-08-18T21:08:31.044 INFO:teuthology.orchestra.run.smithi099.stdout:Recommended packages: 2024-08-18T21:08:31.044 INFO:teuthology.orchestra.run.smithi099.stdout: thermald 2024-08-18T21:08:31.095 INFO:teuthology.orchestra.run.smithi099.stdout:The following NEW packages will be installed: 2024-08-18T21:08:31.096 INFO:teuthology.orchestra.run.smithi099.stdout: linux-headers-5.15.0-118 linux-headers-5.15.0-118-generic 2024-08-18T21:08:31.096 INFO:teuthology.orchestra.run.smithi099.stdout: linux-image-5.15.0-118-generic linux-modules-5.15.0-118-generic 2024-08-18T21:08:31.096 INFO:teuthology.orchestra.run.smithi099.stdout: linux-modules-extra-5.15.0-118-generic 2024-08-18T21:08:31.097 INFO:teuthology.orchestra.run.smithi099.stdout:The following packages will be upgraded: 2024-08-18T21:08:31.098 INFO:teuthology.orchestra.run.smithi099.stdout: linux-generic linux-headers-generic linux-image-generic 2024-08-18T21:08:31.499 INFO:teuthology.orchestra.run.smithi099.stdout:3 upgraded, 5 newly installed, 0 to remove and 333 not upgraded. 2024-08-18T21:08:31.499 INFO:teuthology.orchestra.run.smithi099.stdout:Need to get 113 MB of archives. 2024-08-18T21:08:31.499 INFO:teuthology.orchestra.run.smithi099.stdout:After this operation, 583 MB of additional disk space will be used. 2024-08-18T21:08:31.500 INFO:teuthology.orchestra.run.smithi099.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-118-generic amd64 5.15.0-118.128 [22.6 MB] 2024-08-18T21:08:31.939 INFO:teuthology.orchestra.run.smithi099.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-118-generic amd64 5.15.0-118.128 [11.5 MB] 2024-08-18T21:08:32.051 INFO:teuthology.orchestra.run.smithi099.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-118-generic amd64 5.15.0-118.128 [64.0 MB] 2024-08-18T21:08:32.831 INFO:teuthology.orchestra.run.smithi099.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.118.118 [1,694 B] 2024-08-18T21:08:32.832 INFO:teuthology.orchestra.run.smithi099.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.118.118 [2,466 B] 2024-08-18T21:08:32.832 INFO:teuthology.orchestra.run.smithi099.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-118 all 5.15.0-118.128 [12.4 MB] 2024-08-18T21:08:32.954 INFO:teuthology.orchestra.run.smithi099.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-118-generic amd64 5.15.0-118.128 [2,876 kB] 2024-08-18T21:08:32.977 INFO:teuthology.orchestra.run.smithi099.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.118.118 [2,322 B] 2024-08-18T21:08:33.351 INFO:teuthology.orchestra.run.smithi099.stdout:Fetched 113 MB in 2s (60.6 MB/s) 2024-08-18T21:08:33.540 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package linux-modules-5.15.0-118-generic. 2024-08-18T21:08:35.579 INFO:teuthology.orchestra.run.smithi099.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-08-18T21:08:35.583 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../0-linux-modules-5.15.0-118-generic_5.15.0-118.128_amd64.deb ... 2024-08-18T21:08:35.654 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking linux-modules-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-18T21:08:38.079 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package linux-image-5.15.0-118-generic. 2024-08-18T21:08:38.100 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../1-linux-image-5.15.0-118-generic_5.15.0-118.128_amd64.deb ... 2024-08-18T21:08:38.200 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking linux-image-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-18T21:08:38.640 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-118-generic. 2024-08-18T21:08:38.661 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-118-generic_5.15.0-118.128_amd64.deb ... 2024-08-18T21:08:38.690 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking linux-modules-extra-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-18T21:08:45.248 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../3-linux-generic_5.15.0.118.118_amd64.deb ... 2024-08-18T21:08:45.364 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking linux-generic (5.15.0.118.118) over (5.15.0.56.54) ... 2024-08-18T21:08:45.655 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.118.118_amd64.deb ... 2024-08-18T21:08:45.751 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking linux-image-generic (5.15.0.118.118) over (5.15.0.56.54) ... 2024-08-18T21:08:46.070 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package linux-headers-5.15.0-118. 2024-08-18T21:08:46.089 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../5-linux-headers-5.15.0-118_5.15.0-118.128_all.deb ... 2024-08-18T21:08:46.128 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking linux-headers-5.15.0-118 (5.15.0-118.128) ... 2024-08-18T21:08:51.766 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package linux-headers-5.15.0-118-generic. 2024-08-18T21:08:51.800 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../6-linux-headers-5.15.0-118-generic_5.15.0-118.128_amd64.deb ... 2024-08-18T21:08:51.830 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking linux-headers-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-18T21:08:54.656 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.118.118_amd64.deb ... 2024-08-18T21:08:54.839 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking linux-headers-generic (5.15.0.118.118) over (5.15.0.56.54) ... 2024-08-18T21:08:55.173 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up linux-headers-5.15.0-118 (5.15.0-118.128) ... 2024-08-18T21:08:55.277 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up linux-headers-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-18T21:08:55.378 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up linux-headers-generic (5.15.0.118.118) ... 2024-08-18T21:08:55.507 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up linux-image-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-18T21:08:56.826 INFO:teuthology.orchestra.run.smithi099.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-08-18T21:08:56.826 INFO:teuthology.orchestra.run.smithi099.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-08-18T21:08:56.826 INFO:teuthology.orchestra.run.smithi099.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-118-generic 2024-08-18T21:08:56.826 INFO:teuthology.orchestra.run.smithi099.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-118-generic 2024-08-18T21:08:56.975 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up linux-modules-extra-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-18T21:08:59.077 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up linux-modules-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-18T21:09:00.486 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up linux-image-generic (5.15.0.118.118) ... 2024-08-18T21:09:00.612 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up linux-generic (5.15.0.118.118) ... 2024-08-18T21:09:00.738 INFO:teuthology.orchestra.run.smithi099.stdout:Processing triggers for linux-image-5.15.0-118-generic (5.15.0-118.128) ... 2024-08-18T21:09:00.805 INFO:teuthology.orchestra.run.smithi099.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-08-18T21:09:00.805 INFO:teuthology.orchestra.run.smithi099.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-118-generic 2024-08-18T21:09:15.352 INFO:teuthology.orchestra.run.smithi099.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-08-18T21:09:15.352 INFO:teuthology.orchestra.run.smithi099.stdout:Sourcing file `/etc/default/grub' 2024-08-18T21:09:15.357 INFO:teuthology.orchestra.run.smithi099.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-08-18T21:09:15.379 INFO:teuthology.orchestra.run.smithi099.stdout:Generating grub configuration file ... 2024-08-18T21:09:15.638 INFO:teuthology.orchestra.run.smithi099.stdout:Found linux image: /boot/vmlinuz-5.15.0-118-generic 2024-08-18T21:09:15.658 INFO:teuthology.orchestra.run.smithi099.stdout:Found initrd image: /boot/initrd.img-5.15.0-118-generic 2024-08-18T21:09:15.947 INFO:teuthology.orchestra.run.smithi099.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-08-18T21:09:15.952 INFO:teuthology.orchestra.run.smithi099.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-08-18T21:09:16.023 INFO:teuthology.orchestra.run.smithi099.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-08-18T21:09:16.029 INFO:teuthology.orchestra.run.smithi099.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-08-18T21:09:16.189 INFO:teuthology.orchestra.run.smithi099.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-08-18T21:09:16.189 INFO:teuthology.orchestra.run.smithi099.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-08-18T21:09:16.190 INFO:teuthology.orchestra.run.smithi099.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-08-18T21:09:16.221 INFO:teuthology.orchestra.run.smithi099.stdout:done 2024-08-18T21:09:16.697 DEBUG:teuthology.orchestra.run.smithi099:> dpkg -s linux-image-generic 2024-08-18T21:09:16.716 INFO:teuthology.orchestra.run.smithi099.stdout:Package: linux-image-generic 2024-08-18T21:09:16.716 INFO:teuthology.orchestra.run.smithi099.stdout:Status: install ok installed 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout:Priority: optional 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout:Section: kernel 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout:Installed-Size: 21 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout:Maintainer: Ubuntu Kernel Team 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout:Architecture: amd64 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout:Source: linux-meta 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout:Version: 5.15.0.118.118 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-118), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout:Depends: linux-image-5.15.0-118-generic, linux-modules-extra-5.15.0-118-generic, linux-firmware, intel-microcode, amd64-microcode 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout:Recommends: thermald 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout:Description: Generic Linux kernel image 2024-08-18T21:09:16.717 INFO:teuthology.orchestra.run.smithi099.stdout: This package will always depend on the latest generic kernel image 2024-08-18T21:09:16.718 INFO:teuthology.orchestra.run.smithi099.stdout: available. 2024-08-18T21:09:16.718 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-118-generic 2024-08-18T21:09:16.718 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-18T21:09:16.718 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-18T21:09:16.718 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-18T21:09:16.718 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi099.front.sepia.ceph.com, path=None, version=distro) 2024-08-18T21:09:16.718 DEBUG:teuthology.orchestra.run.smithi099:> sudo apt-get clean 2024-08-18T21:09:16.876 DEBUG:teuthology.orchestra.run.smithi099:> sudo apt-get update 2024-08-18T21:09:16.995 INFO:teuthology.orchestra.run.smithi099.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-08-18T21:09:16.995 INFO:teuthology.orchestra.run.smithi099.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-18T21:09:17.005 INFO:teuthology.orchestra.run.smithi099.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-08-18T21:09:17.029 INFO:teuthology.orchestra.run.smithi099.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-08-18T21:09:18.351 INFO:teuthology.orchestra.run.smithi099.stdout:Reading package lists... 2024-08-18T21:09:18.369 DEBUG:teuthology.orchestra.run.smithi099:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-08-18T21:09:18.422 INFO:teuthology.orchestra.run.smithi099.stdout:Reading package lists... 2024-08-18T21:09:18.606 INFO:teuthology.orchestra.run.smithi099.stdout:Building dependency tree... 2024-08-18T21:09:18.606 INFO:teuthology.orchestra.run.smithi099.stdout:Reading state information... 2024-08-18T21:09:18.771 INFO:teuthology.orchestra.run.smithi099.stdout:linux-image-generic is already the newest version (5.15.0.118.118). 2024-08-18T21:09:18.771 INFO:teuthology.orchestra.run.smithi099.stdout:The following packages were automatically installed and are no longer required: 2024-08-18T21:09:18.771 INFO:teuthology.orchestra.run.smithi099.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-18T21:09:18.771 INFO:teuthology.orchestra.run.smithi099.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-18T21:09:18.771 INFO:teuthology.orchestra.run.smithi099.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-18T21:09:18.772 INFO:teuthology.orchestra.run.smithi099.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-08-18T21:09:18.772 INFO:teuthology.orchestra.run.smithi099.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-08-18T21:09:18.772 INFO:teuthology.orchestra.run.smithi099.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-18T21:09:18.811 INFO:teuthology.orchestra.run.smithi099.stdout:0 upgraded, 0 newly installed, 0 to remove and 333 not upgraded. 2024-08-18T21:09:18.813 DEBUG:teuthology.orchestra.run.smithi099:> dpkg -s linux-image-generic 2024-08-18T21:09:18.871 INFO:teuthology.orchestra.run.smithi099.stdout:Package: linux-image-generic 2024-08-18T21:09:18.872 INFO:teuthology.orchestra.run.smithi099.stdout:Status: install ok installed 2024-08-18T21:09:18.872 INFO:teuthology.orchestra.run.smithi099.stdout:Priority: optional 2024-08-18T21:09:18.872 INFO:teuthology.orchestra.run.smithi099.stdout:Section: kernel 2024-08-18T21:09:18.872 INFO:teuthology.orchestra.run.smithi099.stdout:Installed-Size: 21 2024-08-18T21:09:18.872 INFO:teuthology.orchestra.run.smithi099.stdout:Maintainer: Ubuntu Kernel Team 2024-08-18T21:09:18.872 INFO:teuthology.orchestra.run.smithi099.stdout:Architecture: amd64 2024-08-18T21:09:18.872 INFO:teuthology.orchestra.run.smithi099.stdout:Source: linux-meta 2024-08-18T21:09:18.872 INFO:teuthology.orchestra.run.smithi099.stdout:Version: 5.15.0.118.118 2024-08-18T21:09:18.872 INFO:teuthology.orchestra.run.smithi099.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-118), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-08-18T21:09:18.872 INFO:teuthology.orchestra.run.smithi099.stdout:Depends: linux-image-5.15.0-118-generic, linux-modules-extra-5.15.0-118-generic, linux-firmware, intel-microcode, amd64-microcode 2024-08-18T21:09:18.872 INFO:teuthology.orchestra.run.smithi099.stdout:Recommends: thermald 2024-08-18T21:09:18.873 INFO:teuthology.orchestra.run.smithi099.stdout:Description: Generic Linux kernel image 2024-08-18T21:09:18.873 INFO:teuthology.orchestra.run.smithi099.stdout: This package will always depend on the latest generic kernel image 2024-08-18T21:09:18.873 INFO:teuthology.orchestra.run.smithi099.stdout: available. 2024-08-18T21:09:18.873 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2024-08-18T21:09:18.917 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.slrYIE0vza 2024-08-18T21:09:18.917 DEBUG:teuthology.orchestra.run.smithi099:> sudo cp /boot/grub/grub.cfg /tmp/tmp.slrYIE0vza 2024-08-18T21:09:18.983 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.slrYIE0vza 2024-08-18T21:09:19.106 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.slrYIE0vza is 10KB 2024-08-18T21:09:19.119 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.slrYIE0vza 2024-08-18T21:09:19.125 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-08-18T21:09:19.181 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-08-18T21:09:19.182 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-08-18T21:09:19.182 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-08-18T21:09:19.258 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-118-generic 2024-08-18T21:09:19.258 DEBUG:teuthology.orchestra.run.smithi099:> sudo update-grub 2024-08-18T21:09:20.476 INFO:teuthology.orchestra.run.smithi099.stderr:Sourcing file `/etc/default/grub' 2024-08-18T21:09:20.477 INFO:teuthology.orchestra.run.smithi099.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-08-18T21:09:20.479 INFO:teuthology.orchestra.run.smithi099.stderr:Generating grub configuration file ... 2024-08-18T21:09:20.666 INFO:teuthology.orchestra.run.smithi099.stderr:Found linux image: /boot/vmlinuz-5.15.0-118-generic 2024-08-18T21:09:20.673 INFO:teuthology.orchestra.run.smithi099.stderr:Found initrd image: /boot/initrd.img-5.15.0-118-generic 2024-08-18T21:09:20.907 INFO:teuthology.orchestra.run.smithi099.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-08-18T21:09:20.912 INFO:teuthology.orchestra.run.smithi099.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-08-18T21:09:20.952 INFO:teuthology.orchestra.run.smithi099.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-08-18T21:09:20.957 INFO:teuthology.orchestra.run.smithi099.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-08-18T21:09:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-08-18T21:09:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-08-18T21:09:21.073 INFO:teuthology.orchestra.run.smithi099.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-08-18T21:09:21.091 INFO:teuthology.orchestra.run.smithi099.stderr:done 2024-08-18T21:09:21.100 DEBUG:teuthology.orchestra.run.smithi099:> sudo shutdown -r now 2024-08-18T21:09:51.130 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-18T21:09:51.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-08-18T21:09:51.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T21:10:25.759 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.99 2024-08-18T21:10:34.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-08-18T21:10:34.768 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T21:11:06.515 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-08-18T21:11:07.450 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-08-18T21:11:07.450 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-118-generic"... 2024-08-18T21:11:07.450 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-08-18T21:11:07.455 INFO:teuthology.orchestra.run.smithi099.stdout:5.15.0-118-generic 2024-08-18T21:11:07.456 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-118-generic vs 5.15.0-118-generic 2024-08-18T21:11:07.456 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-18T21:11:07.456 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-08-18T21:11:08.457 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-18T21:11:08.458 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-18T21:11:08.534 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2024-08-18T21:11:08.562 DEBUG:teuthology.parallel:result is None 2024-08-18T21:11:08.565 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-18T21:11:08.573 INFO:teuthology.task.internal:Creating test directory... 2024-08-18T21:11:08.573 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-18T21:11:08.579 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-18T21:11:08.585 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-18T21:11:08.600 INFO:teuthology.task.internal:Creating archive directory... 2024-08-18T21:11:08.601 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-18T21:11:08.643 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-18T21:11:08.654 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-18T21:11:08.654 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-18T21:11:08.700 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:11:08.707 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:11:08.710 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-18T21:11:08.718 INFO:teuthology.task.internal:Configuring sudo... 2024-08-18T21:11:08.718 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-18T21:11:08.765 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-18T21:11:08.780 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-18T21:11:08.780 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-18T21:11:08.812 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-18T21:11:08.861 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-18T21:11:08.909 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-08-18T21:11:08.909 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-18T21:11:08.965 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2024-08-18T21:11:09.046 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-18T21:11:09.058 INFO:teuthology.task.internal:Starting timer... 2024-08-18T21:11:09.058 INFO:teuthology.run_tasks:Running task pcp... 2024-08-18T21:11:09.074 INFO:teuthology.run_tasks:Running task selinux... 2024-08-18T21:11:09.087 DEBUG:teuthology.task.selinux:Excluding smithi099: OS 'ubuntu' does not support SELinux 2024-08-18T21:11:09.087 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-18T21:11:09.087 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-08-18T21:11:09.088 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-18T21:11:09.088 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-18T21:11:09.097 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-18T21:11:09.109 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-18T21:11:09.110 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi099.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-18T21:18:37.010 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi099.front.sepia.ceph.com')] 2024-08-18T21:18:37.012 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-08-18T21:18:37.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-18T21:18:37.096 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-08-18T21:18:37.170 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-08-18T21:18:37.170 INFO:teuthology.run_tasks:Running task clock... 2024-08-18T21:18:37.182 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-18T21:18:37.182 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-18T21:18:37.183 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-18T21:18:37.238 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-08-18T21:18:37.238 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: Command line: ntpd -gq 2024-08-18T21:18:37.238 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: ---------------------------------------------------- 2024-08-18T21:18:37.238 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: ntp-4 is maintained by Network Time Foundation, 2024-08-18T21:18:37.238 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-08-18T21:18:37.238 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: corporation. Support and training for ntp-4 are 2024-08-18T21:18:37.238 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: available at https://www.nwtime.org/support 2024-08-18T21:18:37.238 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: ---------------------------------------------------- 2024-08-18T21:18:37.238 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: proto: precision = 0.043 usec (-24) 2024-08-18T21:18:37.239 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: basedate set to 2022-02-04 2024-08-18T21:18:37.239 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: gps base set to 2022-02-06 (week 2196) 2024-08-18T21:18:37.239 INFO:teuthology.orchestra.run.smithi099.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-08-18T21:18:37.240 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-08-18T21:18:37.240 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: restrict ::: KOD does nothing without LIMITED. 2024-08-18T21:18:37.240 INFO:teuthology.orchestra.run.smithi099.stderr:restrict ::: KOD does nothing without LIMITED. 2024-08-18T21:18:37.241 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: Listen and drop on 0 v6wildcard [::]:123 2024-08-18T21:18:37.241 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-08-18T21:18:37.241 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: Listen normally on 2 lo 127.0.0.1:123 2024-08-18T21:18:37.241 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: Listen normally on 3 enp3s0f1 172.21.15.99:123 2024-08-18T21:18:37.241 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: Listen normally on 4 lo [::1]:123 2024-08-18T21:18:37.241 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d803%5]:123 2024-08-18T21:18:37.241 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:37 ntpd[18373]: Listening on routing socket on fd #22 for interface updates 2024-08-18T21:18:39.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:39 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:39.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:39 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:39.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:39 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:39.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:39 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:39.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:39 ntpd[18373]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-18T21:18:39.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:39 ntpd[18373]: can't open /var/log/ntpstats/peerstats.20240818: Permission denied 2024-08-18T21:18:40.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:40 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:40.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:40 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:40.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:40 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:40.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:40 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:40.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:40 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:40.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:40 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:40.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:40 ntpd[18373]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-18T21:18:40.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:40 ntpd[18373]: can't open /var/log/ntpstats/peerstats.20240818: Permission denied 2024-08-18T21:18:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-18T21:18:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: can't open /var/log/ntpstats/peerstats.20240818: Permission denied 2024-08-18T21:18:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:41.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-18T21:18:41.240 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:41 ntpd[18373]: can't open /var/log/ntpstats/peerstats.20240818: Permission denied 2024-08-18T21:18:42.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:42 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:42.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:42 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:42.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:42 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:42.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:42 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:42.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:42 ntpd[18373]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-18T21:18:42.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:42 ntpd[18373]: can't open /var/log/ntpstats/peerstats.20240818: Permission denied 2024-08-18T21:18:43.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:43.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:43.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:43.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:43.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:43.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:43.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-18T21:18:43.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: can't open /var/log/ntpstats/peerstats.20240818: Permission denied 2024-08-18T21:18:43.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:43.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:43.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-18T21:18:43.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:43 ntpd[18373]: can't open /var/log/ntpstats/peerstats.20240818: Permission denied 2024-08-18T21:18:44.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:44 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:44.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:44 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:44.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:44 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:44.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:44 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:44.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:44 ntpd[18373]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-18T21:18:44.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:44 ntpd[18373]: can't open /var/log/ntpstats/peerstats.20240818: Permission denied 2024-08-18T21:18:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:45 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:45 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:45.238 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:45 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:45 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:45 ntpd[18373]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-08-18T21:18:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:45 ntpd[18373]: can't open /var/log/ntpstats/rawstats.20240818: Permission denied 2024-08-18T21:18:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:45 ntpd[18373]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-08-18T21:18:45.239 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:45 ntpd[18373]: can't open /var/log/ntpstats/peerstats.20240818: Permission denied 2024-08-18T21:18:45.239 INFO:teuthology.orchestra.run.smithi099.stdout:18 Aug 21:18:45 ntpd[18373]: ntpd: time slew -0.002030 s 2024-08-18T21:18:45.239 INFO:teuthology.orchestra.run.smithi099.stdout:ntpd: time slew -0.002030s 2024-08-18T21:18:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:45 ntpd[18373]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-08-18T21:18:45.240 INFO:teuthology.orchestra.run.smithi099.stderr:18 Aug 21:18:45 ntpd[18373]: can't open /var/log/ntpstats/loopstats.20240818: Permission denied 2024-08-18T21:18:45.355 INFO:teuthology.orchestra.run.smithi099.stdout: remote refid st t when poll reach delay offset jitter 2024-08-18T21:18:45.355 INFO:teuthology.orchestra.run.smithi099.stdout:============================================================================== 2024-08-18T21:18:45.356 INFO:teuthology.orchestra.run.smithi099.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-18T21:18:45.356 INFO:teuthology.orchestra.run.smithi099.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-18T21:18:45.356 INFO:teuthology.orchestra.run.smithi099.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-18T21:18:45.356 INFO:teuthology.orchestra.run.smithi099.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-08-18T21:18:45.357 INFO:teuthology.run_tasks:Running task install... 2024-08-18T21:18:45.368 DEBUG:teuthology.task.install:project ceph 2024-08-18T21:18:45.368 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'}} 2024-08-18T21:18:45.369 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '797669399b32d2915e64eb6d4b1257ae3c50f80b'} 2024-08-18T21:18:45.369 INFO:teuthology.task.install:Using flavor: default 2024-08-18T21:18:45.374 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-18T21:18:45.374 INFO:teuthology.task.install:extra packages: [] 2024-08-18T21:18:45.374 DEBUG:teuthology.orchestra.run.smithi099:> sudo apt-key list | grep Ceph 2024-08-18T21:18:45.457 INFO:teuthology.orchestra.run.smithi099.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-08-18T21:18:45.487 INFO:teuthology.orchestra.run.smithi099.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-08-18T21:18:45.487 INFO:teuthology.orchestra.run.smithi099.stdout:uid [ unknown] Ceph.com (release key) 2024-08-18T21:18:45.487 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-08-18T21:18:45.488 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-18T21:18:45.640 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default/ 2024-08-18T21:18:45.640 INFO:teuthology.task.install.deb:Package version is 19.0.0-5847-g79766939-1jammy 2024-08-18T21:18:45.756 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-08-18T21:18:45.757 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-08-18T21:18:45.771 DEBUG:teuthology.orchestra.run.smithi099:> sudo apt-get update 2024-08-18T21:18:45.962 INFO:teuthology.orchestra.run.smithi099.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-08-18T21:18:45.994 INFO:teuthology.orchestra.run.smithi099.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy InRelease 2024-08-18T21:18:46.017 INFO:teuthology.orchestra.run.smithi099.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy Release [7,005 B] 2024-08-18T21:18:46.038 INFO:teuthology.orchestra.run.smithi099.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy Release.gpg 2024-08-18T21:18:46.060 INFO:teuthology.orchestra.run.smithi099.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-08-18T21:18:46.066 INFO:teuthology.orchestra.run.smithi099.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-08-18T21:18:46.087 INFO:teuthology.orchestra.run.smithi099.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,422 B] 2024-08-18T21:18:46.142 INFO:teuthology.orchestra.run.smithi099.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-08-18T21:18:46.224 INFO:teuthology.orchestra.run.smithi099.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-08-18T21:18:46.363 INFO:teuthology.orchestra.run.smithi099.stdout:Fetched 28.6 kB in 0s (69.9 kB/s) 2024-08-18T21:18:47.405 INFO:teuthology.orchestra.run.smithi099.stdout:Reading package lists... 2024-08-18T21:18:47.422 DEBUG:teuthology.orchestra.run.smithi099:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-5847-g79766939-1jammy cephadm=19.0.0-5847-g79766939-1jammy ceph-mds=19.0.0-5847-g79766939-1jammy ceph-mgr=19.0.0-5847-g79766939-1jammy ceph-common=19.0.0-5847-g79766939-1jammy ceph-fuse=19.0.0-5847-g79766939-1jammy ceph-test=19.0.0-5847-g79766939-1jammy ceph-volume=19.0.0-5847-g79766939-1jammy radosgw=19.0.0-5847-g79766939-1jammy python3-rados=19.0.0-5847-g79766939-1jammy python3-rgw=19.0.0-5847-g79766939-1jammy python3-cephfs=19.0.0-5847-g79766939-1jammy python3-rbd=19.0.0-5847-g79766939-1jammy libcephfs2=19.0.0-5847-g79766939-1jammy libcephfs-dev=19.0.0-5847-g79766939-1jammy librados2=19.0.0-5847-g79766939-1jammy librbd1=19.0.0-5847-g79766939-1jammy rbd-fuse=19.0.0-5847-g79766939-1jammy 2024-08-18T21:18:47.475 INFO:teuthology.orchestra.run.smithi099.stdout:Reading package lists... 2024-08-18T21:18:47.659 INFO:teuthology.orchestra.run.smithi099.stdout:Building dependency tree... 2024-08-18T21:18:47.659 INFO:teuthology.orchestra.run.smithi099.stdout:Reading state information... 2024-08-18T21:18:47.846 INFO:teuthology.orchestra.run.smithi099.stdout:The following packages were automatically installed and are no longer required: 2024-08-18T21:18:47.846 INFO:teuthology.orchestra.run.smithi099.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-08-18T21:18:47.846 INFO:teuthology.orchestra.run.smithi099.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-08-18T21:18:47.846 INFO:teuthology.orchestra.run.smithi099.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-08-18T21:18:47.846 INFO:teuthology.orchestra.run.smithi099.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-08-18T21:18:47.848 INFO:teuthology.orchestra.run.smithi099.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-08-18T21:18:47.849 INFO:teuthology.orchestra.run.smithi099.stdout:Use 'sudo apt autoremove' to remove them. 2024-08-18T21:18:47.849 INFO:teuthology.orchestra.run.smithi099.stdout:The following additional packages will be installed: 2024-08-18T21:18:47.849 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-08-18T21:18:47.849 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-08-18T21:18:47.849 INFO:teuthology.orchestra.run.smithi099.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-08-18T21:18:47.849 INFO:teuthology.orchestra.run.smithi099.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-08-18T21:18:47.849 INFO:teuthology.orchestra.run.smithi099.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-08-18T21:18:47.850 INFO:teuthology.orchestra.run.smithi099.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-08-18T21:18:47.850 INFO:teuthology.orchestra.run.smithi099.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-08-18T21:18:47.850 INFO:teuthology.orchestra.run.smithi099.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-08-18T21:18:47.850 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-08-18T21:18:47.850 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-08-18T21:18:47.850 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-08-18T21:18:47.850 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-08-18T21:18:47.850 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-08-18T21:18:47.850 INFO:teuthology.orchestra.run.smithi099.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-08-18T21:18:47.850 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-08-18T21:18:47.850 INFO:teuthology.orchestra.run.smithi099.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-08-18T21:18:47.851 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-08-18T21:18:47.851 INFO:teuthology.orchestra.run.smithi099.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-08-18T21:18:47.851 INFO:teuthology.orchestra.run.smithi099.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-08-18T21:18:47.851 INFO:teuthology.orchestra.run.smithi099.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-08-18T21:18:47.851 INFO:teuthology.orchestra.run.smithi099.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-08-18T21:18:47.851 INFO:teuthology.orchestra.run.smithi099.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-08-18T21:18:47.851 INFO:teuthology.orchestra.run.smithi099.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-08-18T21:18:47.851 INFO:teuthology.orchestra.run.smithi099.stdout: xmlstarlet zip 2024-08-18T21:18:47.852 INFO:teuthology.orchestra.run.smithi099.stdout:Suggested packages: 2024-08-18T21:18:47.852 INFO:teuthology.orchestra.run.smithi099.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-08-18T21:18:47.852 INFO:teuthology.orchestra.run.smithi099.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-08-18T21:18:47.852 INFO:teuthology.orchestra.run.smithi099.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-08-18T21:18:47.852 INFO:teuthology.orchestra.run.smithi099.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-08-18T21:18:47.852 INFO:teuthology.orchestra.run.smithi099.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-08-18T21:18:47.852 INFO:teuthology.orchestra.run.smithi099.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-08-18T21:18:47.852 INFO:teuthology.orchestra.run.smithi099.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-08-18T21:18:47.852 INFO:teuthology.orchestra.run.smithi099.stdout:Recommended packages: 2024-08-18T21:18:47.852 INFO:teuthology.orchestra.run.smithi099.stdout: btrfs-tools 2024-08-18T21:18:47.953 INFO:teuthology.orchestra.run.smithi099.stdout:The following NEW packages will be installed: 2024-08-18T21:18:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-08-18T21:18:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-08-18T21:18:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-08-18T21:18:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-08-18T21:18:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-08-18T21:18:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-08-18T21:18:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-08-18T21:18:47.953 INFO:teuthology.orchestra.run.smithi099.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-08-18T21:18:47.954 INFO:teuthology.orchestra.run.smithi099.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-08-18T21:18:47.955 INFO:teuthology.orchestra.run.smithi099.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-08-18T21:18:47.955 INFO:teuthology.orchestra.run.smithi099.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-08-18T21:18:47.955 INFO:teuthology.orchestra.run.smithi099.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-08-18T21:18:47.955 INFO:teuthology.orchestra.run.smithi099.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-08-18T21:18:47.955 INFO:teuthology.orchestra.run.smithi099.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-08-18T21:18:48.098 INFO:teuthology.orchestra.run.smithi099.stdout:0 upgraded, 118 newly installed, 0 to remove and 320 not upgraded. 2024-08-18T21:18:48.098 INFO:teuthology.orchestra.run.smithi099.stdout:Need to get 251 MB of archives. 2024-08-18T21:18:48.098 INFO:teuthology.orchestra.run.smithi099.stdout:After this operation, 932 MB of additional disk space will be used. 2024-08-18T21:18:48.098 INFO:teuthology.orchestra.run.smithi099.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-5847-g79766939-1jammy [3,632 kB] 2024-08-18T21:18:48.142 INFO:teuthology.orchestra.run.smithi099.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-08-18T21:18:48.294 INFO:teuthology.orchestra.run.smithi099.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-5847-g79766939-1jammy [3,313 kB] 2024-08-18T21:18:48.351 INFO:teuthology.orchestra.run.smithi099.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-5847-g79766939-1jammy [721 kB] 2024-08-18T21:18:48.362 INFO:teuthology.orchestra.run.smithi099.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-5847-g79766939-1jammy [356 kB] 2024-08-18T21:18:48.366 INFO:teuthology.orchestra.run.smithi099.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-5847-g79766939-1jammy [32.8 kB] 2024-08-18T21:18:48.367 INFO:teuthology.orchestra.run.smithi099.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-5847-g79766939-1jammy [180 kB] 2024-08-18T21:18:48.369 INFO:teuthology.orchestra.run.smithi099.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-5847-g79766939-1jammy [68.9 kB] 2024-08-18T21:18:48.370 INFO:teuthology.orchestra.run.smithi099.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-5847-g79766939-1jammy [338 kB] 2024-08-18T21:18:48.379 INFO:teuthology.orchestra.run.smithi099.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-5847-g79766939-1jammy [8,017 kB] 2024-08-18T21:18:48.504 INFO:teuthology.orchestra.run.smithi099.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-5847-g79766939-1jammy [112 kB] 2024-08-18T21:18:48.505 INFO:teuthology.orchestra.run.smithi099.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-5847-g79766939-1jammy [472 kB] 2024-08-18T21:18:48.512 INFO:teuthology.orchestra.run.smithi099.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-5847-g79766939-1jammy [28.3 MB] 2024-08-18T21:18:48.565 INFO:teuthology.orchestra.run.smithi099.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-08-18T21:18:48.577 INFO:teuthology.orchestra.run.smithi099.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-08-18T21:18:48.658 INFO:teuthology.orchestra.run.smithi099.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-08-18T21:18:48.890 INFO:teuthology.orchestra.run.smithi099.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-08-18T21:18:48.897 INFO:teuthology.orchestra.run.smithi099.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-08-18T21:18:48.927 INFO:teuthology.orchestra.run.smithi099.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-08-18T21:18:48.936 INFO:teuthology.orchestra.run.smithi099.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-08-18T21:18:48.937 INFO:teuthology.orchestra.run.smithi099.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-08-18T21:18:48.938 INFO:teuthology.orchestra.run.smithi099.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-08-18T21:18:48.956 INFO:teuthology.orchestra.run.smithi099.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-08-18T21:18:48.960 INFO:teuthology.orchestra.run.smithi099.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-08-18T21:18:48.964 INFO:teuthology.orchestra.run.smithi099.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-08-18T21:18:48.969 INFO:teuthology.orchestra.run.smithi099.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-08-18T21:18:49.049 INFO:teuthology.orchestra.run.smithi099.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-08-18T21:18:49.123 INFO:teuthology.orchestra.run.smithi099.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-08-18T21:18:49.123 INFO:teuthology.orchestra.run.smithi099.stdout:Get:29 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-5847-g79766939-1jammy [5,198 kB] 2024-08-18T21:18:49.127 INFO:teuthology.orchestra.run.smithi099.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-08-18T21:18:49.130 INFO:teuthology.orchestra.run.smithi099.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-08-18T21:18:49.132 INFO:teuthology.orchestra.run.smithi099.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-08-18T21:18:49.132 INFO:teuthology.orchestra.run.smithi099.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-08-18T21:18:49.135 INFO:teuthology.orchestra.run.smithi099.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-08-18T21:18:49.135 INFO:teuthology.orchestra.run.smithi099.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-08-18T21:18:49.136 INFO:teuthology.orchestra.run.smithi099.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-08-18T21:18:49.136 INFO:teuthology.orchestra.run.smithi099.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-08-18T21:18:49.188 INFO:teuthology.orchestra.run.smithi099.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-5847-g79766939-1jammy [247 kB] 2024-08-18T21:18:49.190 INFO:teuthology.orchestra.run.smithi099.stdout:Get:39 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-5847-g79766939-1jammy [124 kB] 2024-08-18T21:18:49.192 INFO:teuthology.orchestra.run.smithi099.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-5847-g79766939-1jammy [1,560 kB] 2024-08-18T21:18:49.205 INFO:teuthology.orchestra.run.smithi099.stdout:Get:41 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-5847-g79766939-1jammy [6,293 kB] 2024-08-18T21:18:49.213 INFO:teuthology.orchestra.run.smithi099.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-08-18T21:18:49.219 INFO:teuthology.orchestra.run.smithi099.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-08-18T21:18:49.220 INFO:teuthology.orchestra.run.smithi099.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-08-18T21:18:49.226 INFO:teuthology.orchestra.run.smithi099.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-08-18T21:18:49.227 INFO:teuthology.orchestra.run.smithi099.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-08-18T21:18:49.228 INFO:teuthology.orchestra.run.smithi099.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-08-18T21:18:49.229 INFO:teuthology.orchestra.run.smithi099.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-08-18T21:18:49.231 INFO:teuthology.orchestra.run.smithi099.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-08-18T21:18:49.231 INFO:teuthology.orchestra.run.smithi099.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-08-18T21:18:49.294 INFO:teuthology.orchestra.run.smithi099.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-08-18T21:18:49.297 INFO:teuthology.orchestra.run.smithi099.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-5847-g79766939-1jammy [23.1 MB] 2024-08-18T21:18:49.314 INFO:teuthology.orchestra.run.smithi099.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-08-18T21:18:49.316 INFO:teuthology.orchestra.run.smithi099.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-08-18T21:18:49.317 INFO:teuthology.orchestra.run.smithi099.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-08-18T21:18:49.329 INFO:teuthology.orchestra.run.smithi099.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-08-18T21:18:49.329 INFO:teuthology.orchestra.run.smithi099.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-08-18T21:18:49.331 INFO:teuthology.orchestra.run.smithi099.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-08-18T21:18:49.332 INFO:teuthology.orchestra.run.smithi099.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-08-18T21:18:49.377 INFO:teuthology.orchestra.run.smithi099.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-08-18T21:18:49.382 INFO:teuthology.orchestra.run.smithi099.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-08-18T21:18:49.459 INFO:teuthology.orchestra.run.smithi099.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-08-18T21:18:49.462 INFO:teuthology.orchestra.run.smithi099.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-08-18T21:18:49.464 INFO:teuthology.orchestra.run.smithi099.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-08-18T21:18:49.470 INFO:teuthology.orchestra.run.smithi099.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-08-18T21:18:49.471 INFO:teuthology.orchestra.run.smithi099.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-08-18T21:18:49.473 INFO:teuthology.orchestra.run.smithi099.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-08-18T21:18:49.479 INFO:teuthology.orchestra.run.smithi099.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-08-18T21:18:49.643 INFO:teuthology.orchestra.run.smithi099.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-08-18T21:18:49.645 INFO:teuthology.orchestra.run.smithi099.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-08-18T21:18:49.648 INFO:teuthology.orchestra.run.smithi099.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-08-18T21:18:49.649 INFO:teuthology.orchestra.run.smithi099.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-08-18T21:18:49.651 INFO:teuthology.orchestra.run.smithi099.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-08-18T21:18:49.760 INFO:teuthology.orchestra.run.smithi099.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-08-18T21:18:49.812 INFO:teuthology.orchestra.run.smithi099.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-5847-g79766939-1jammy [14.1 kB] 2024-08-18T21:18:49.812 INFO:teuthology.orchestra.run.smithi099.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-08-18T21:18:49.813 INFO:teuthology.orchestra.run.smithi099.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-08-18T21:18:49.813 INFO:teuthology.orchestra.run.smithi099.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-5847-g79766939-1jammy [880 kB] 2024-08-18T21:18:49.824 INFO:teuthology.orchestra.run.smithi099.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-5847-g79766939-1jammy [2,437 kB] 2024-08-18T21:18:49.840 INFO:teuthology.orchestra.run.smithi099.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-08-18T21:18:49.840 INFO:teuthology.orchestra.run.smithi099.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-08-18T21:18:49.841 INFO:teuthology.orchestra.run.smithi099.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-08-18T21:18:49.842 INFO:teuthology.orchestra.run.smithi099.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-08-18T21:18:49.842 INFO:teuthology.orchestra.run.smithi099.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-08-18T21:18:49.843 INFO:teuthology.orchestra.run.smithi099.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-08-18T21:18:49.857 INFO:teuthology.orchestra.run.smithi099.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-5847-g79766939-1jammy [801 kB] 2024-08-18T21:18:49.866 INFO:teuthology.orchestra.run.smithi099.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-5847-g79766939-1jammy [156 kB] 2024-08-18T21:18:49.868 INFO:teuthology.orchestra.run.smithi099.stdout:Get:88 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-5847-g79766939-1jammy [61.3 MB] 2024-08-18T21:18:49.892 INFO:teuthology.orchestra.run.smithi099.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-08-18T21:18:49.926 INFO:teuthology.orchestra.run.smithi099.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-08-18T21:18:49.934 INFO:teuthology.orchestra.run.smithi099.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-08-18T21:18:49.938 INFO:teuthology.orchestra.run.smithi099.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-08-18T21:18:49.942 INFO:teuthology.orchestra.run.smithi099.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-08-18T21:18:49.943 INFO:teuthology.orchestra.run.smithi099.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-08-18T21:18:49.980 INFO:teuthology.orchestra.run.smithi099.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-08-18T21:18:49.987 INFO:teuthology.orchestra.run.smithi099.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-08-18T21:18:49.990 INFO:teuthology.orchestra.run.smithi099.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-08-18T21:18:49.991 INFO:teuthology.orchestra.run.smithi099.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-08-18T21:18:50.054 INFO:teuthology.orchestra.run.smithi099.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-08-18T21:18:50.056 INFO:teuthology.orchestra.run.smithi099.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-08-18T21:18:50.064 INFO:teuthology.orchestra.run.smithi099.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-08-18T21:18:50.064 INFO:teuthology.orchestra.run.smithi099.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-08-18T21:18:50.067 INFO:teuthology.orchestra.run.smithi099.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-08-18T21:18:50.136 INFO:teuthology.orchestra.run.smithi099.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-08-18T21:18:50.139 INFO:teuthology.orchestra.run.smithi099.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-08-18T21:18:50.142 INFO:teuthology.orchestra.run.smithi099.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-08-18T21:18:50.160 INFO:teuthology.orchestra.run.smithi099.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-08-18T21:18:50.161 INFO:teuthology.orchestra.run.smithi099.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-08-18T21:18:50.217 INFO:teuthology.orchestra.run.smithi099.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-08-18T21:18:50.222 INFO:teuthology.orchestra.run.smithi099.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-08-18T21:18:50.224 INFO:teuthology.orchestra.run.smithi099.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-08-18T21:18:51.142 INFO:teuthology.orchestra.run.smithi099.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-5847-g79766939-1jammy [8,625 kB] 2024-08-18T21:18:51.250 INFO:teuthology.orchestra.run.smithi099.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-5847-g79766939-1jammy [14.2 kB] 2024-08-18T21:18:51.250 INFO:teuthology.orchestra.run.smithi099.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-5847-g79766939-1jammy [55.8 MB] 2024-08-18T21:18:52.433 INFO:teuthology.orchestra.run.smithi099.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-5847-g79766939-1jammy [130 kB] 2024-08-18T21:18:52.435 INFO:teuthology.orchestra.run.smithi099.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-5847-g79766939-1jammy [38.9 kB] 2024-08-18T21:18:52.436 INFO:teuthology.orchestra.run.smithi099.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-5847-g79766939-1jammy [16.0 MB] 2024-08-18T21:18:52.749 INFO:teuthology.orchestra.run.smithi099.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/797669399b32d2915e64eb6d4b1257ae3c50f80b/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-5847-g79766939-1jammy [91.4 kB] 2024-08-18T21:18:53.304 INFO:teuthology.orchestra.run.smithi099.stdout:Fetched 251 MB in 5s (52.4 MB/s) 2024-08-18T21:18:53.469 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-08-18T21:18:53.569 INFO:teuthology.orchestra.run.smithi099.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-08-18T21:18:53.575 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-08-18T21:18:53.668 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-08-18T21:18:53.971 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-08-18T21:18:53.997 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-08-18T21:18:54.030 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-08-18T21:18:54.375 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-08-18T21:18:54.400 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-08-18T21:18:54.441 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-08-18T21:18:54.886 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-08-18T21:18:54.912 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-18T21:18:54.961 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-18T21:18:55.365 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-08-18T21:18:55.391 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-18T21:18:55.431 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-18T21:18:55.843 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-08-18T21:18:55.869 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-08-18T21:18:55.909 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-18T21:18:56.263 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-08-18T21:18:56.289 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-08-18T21:18:56.329 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-08-18T21:18:56.658 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package librados2. 2024-08-18T21:18:56.684 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../007-librados2_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:18:56.724 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking librados2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:18:57.185 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package librbd1. 2024-08-18T21:18:57.212 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../008-librbd1_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:18:57.252 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking librbd1 (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:18:57.713 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libcephfs2. 2024-08-18T21:18:57.740 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:18:57.780 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libcephfs2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:18:58.032 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-rados. 2024-08-18T21:18:58.048 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../010-python3-rados_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:18:58.082 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-rados (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:18:58.402 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-08-18T21:18:58.428 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-18T21:18:58.468 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-ceph-argparse (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:18:58.721 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-cephfs. 2024-08-18T21:18:58.748 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:18:58.788 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-cephfs (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:18:59.082 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-ceph-common. 2024-08-18T21:18:59.097 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-18T21:18:59.132 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-ceph-common (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:18:59.469 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-wcwidth. 2024-08-18T21:18:59.495 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-08-18T21:18:59.535 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-08-18T21:18:59.889 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-prettytable. 2024-08-18T21:18:59.915 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-08-18T21:18:59.955 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-08-18T21:19:00.166 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-rbd. 2024-08-18T21:19:00.182 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:00.224 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-rbd (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:00.578 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-08-18T21:19:00.604 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-08-18T21:19:00.644 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-08-18T21:19:00.905 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-08-18T21:19:00.921 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-08-18T21:19:00.964 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-08-18T21:19:01.225 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-08-18T21:19:01.240 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-08-18T21:19:01.275 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-08-18T21:19:01.587 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package lua5.1. 2024-08-18T21:19:01.613 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-08-18T21:19:01.653 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-08-18T21:19:01.969 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package lua-any. 2024-08-18T21:19:01.982 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-08-18T21:19:02.017 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking lua-any (27ubuntu1) ... 2024-08-18T21:19:02.219 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package zip. 2024-08-18T21:19:02.246 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-08-18T21:19:02.286 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking zip (3.0-12build2) ... 2024-08-18T21:19:02.606 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package unzip. 2024-08-18T21:19:02.633 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-08-18T21:19:02.672 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-08-18T21:19:03.102 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package luarocks. 2024-08-18T21:19:03.128 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-08-18T21:19:03.168 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-08-18T21:19:03.501 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package librgw2. 2024-08-18T21:19:03.516 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../025-librgw2_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:03.551 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking librgw2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:04.170 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-rgw. 2024-08-18T21:19:04.185 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:04.229 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-rgw (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:04.582 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package liboath0:amd64. 2024-08-18T21:19:04.608 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-08-18T21:19:04.657 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-08-18T21:19:04.943 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libradosstriper1. 2024-08-18T21:19:04.969 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:05.001 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libradosstriper1 (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:05.397 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-common. 2024-08-18T21:19:05.423 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../029-ceph-common_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:05.463 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-common (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:07.010 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-base. 2024-08-18T21:19:07.037 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../030-ceph-base_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:07.079 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-base (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:07.623 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-08-18T21:19:07.650 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-08-18T21:19:07.690 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-08-18T21:19:08.010 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-cheroot. 2024-08-18T21:19:08.036 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-08-18T21:19:08.076 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-08-18T21:19:08.354 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-08-18T21:19:08.370 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-08-18T21:19:08.404 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-08-18T21:19:08.716 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-jaraco.text. 2024-08-18T21:19:08.742 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-08-18T21:19:08.782 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-08-18T21:19:09.085 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-08-18T21:19:09.112 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-08-18T21:19:09.152 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-08-18T21:19:09.421 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-tempora. 2024-08-18T21:19:09.448 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-08-18T21:19:09.479 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-08-18T21:19:09.800 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-portend. 2024-08-18T21:19:09.826 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-08-18T21:19:09.866 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-08-18T21:19:10.169 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-08-18T21:19:10.195 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-08-18T21:19:10.236 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-08-18T21:19:10.480 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-cherrypy3. 2024-08-18T21:19:10.495 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-08-18T21:19:10.538 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-08-18T21:19:10.893 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-natsort. 2024-08-18T21:19:10.919 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-08-18T21:19:10.959 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-08-18T21:19:11.270 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-logutils. 2024-08-18T21:19:11.297 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-08-18T21:19:11.337 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-08-18T21:19:11.606 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-markupsafe. 2024-08-18T21:19:11.621 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-08-18T21:19:11.665 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-08-18T21:19:11.918 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-mako. 2024-08-18T21:19:11.944 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-08-18T21:19:11.984 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-08-18T21:19:12.305 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-simplegeneric. 2024-08-18T21:19:12.331 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-08-18T21:19:12.371 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-08-18T21:19:12.674 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-singledispatch. 2024-08-18T21:19:12.701 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-08-18T21:19:12.741 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-08-18T21:19:13.019 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-webob. 2024-08-18T21:19:13.045 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-08-18T21:19:13.102 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-08-18T21:19:13.480 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-waitress. 2024-08-18T21:19:13.506 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-08-18T21:19:13.550 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-08-18T21:19:13.825 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-tempita. 2024-08-18T21:19:13.851 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-08-18T21:19:13.883 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-08-18T21:19:14.178 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-paste. 2024-08-18T21:19:14.204 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-08-18T21:19:14.244 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-08-18T21:19:14.598 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-08-18T21:19:14.613 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-08-18T21:19:14.656 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-08-18T21:19:14.926 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-pastedeploy. 2024-08-18T21:19:14.952 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-08-18T21:19:14.992 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-08-18T21:19:15.245 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-webtest. 2024-08-18T21:19:15.272 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-08-18T21:19:15.312 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-08-18T21:19:15.640 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-pecan. 2024-08-18T21:19:15.666 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-08-18T21:19:15.706 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-08-18T21:19:16.053 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-certifi. 2024-08-18T21:19:16.080 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-08-18T21:19:16.142 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-08-18T21:19:16.498 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-idna. 2024-08-18T21:19:16.524 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-08-18T21:19:16.564 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-08-18T21:19:16.834 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-urllib3. 2024-08-18T21:19:16.861 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-08-18T21:19:16.894 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-08-18T21:19:17.196 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-requests. 2024-08-18T21:19:17.222 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-08-18T21:19:17.254 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-08-18T21:19:17.524 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-werkzeug. 2024-08-18T21:19:17.550 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-08-18T21:19:17.599 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-08-18T21:19:17.877 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-08-18T21:19:17.893 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-18T21:19:17.927 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-mgr-modules-core (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:18.299 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-08-18T21:19:18.326 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:18.365 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:18.652 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-bcrypt. 2024-08-18T21:19:18.668 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-08-18T21:19:18.701 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-08-18T21:19:18.971 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-openssl. 2024-08-18T21:19:18.997 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-08-18T21:19:19.037 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-08-18T21:19:19.391 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-mgr. 2024-08-18T21:19:19.417 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:19.457 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-mgr (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:19.878 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-mon. 2024-08-18T21:19:19.905 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:19.944 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-mon (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:20.522 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-osd. 2024-08-18T21:19:20.550 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:20.589 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-osd (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:22.676 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph. 2024-08-18T21:19:22.691 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../066-ceph_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:22.734 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:23.021 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-fuse. 2024-08-18T21:19:23.047 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:23.087 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-fuse (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:23.424 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-mds. 2024-08-18T21:19:23.450 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:23.490 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-mds (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:23.893 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package cephadm. 2024-08-18T21:19:23.920 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../069-cephadm_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:23.960 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking cephadm (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:24.279 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-asyncssh. 2024-08-18T21:19:24.306 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-08-18T21:19:24.346 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-08-18T21:19:24.691 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python-babel-localedata. 2024-08-18T21:19:24.717 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-08-18T21:19:24.757 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-08-18T21:19:25.770 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-babel. 2024-08-18T21:19:25.798 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-08-18T21:19:25.828 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-08-18T21:19:26.098 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-jinja2. 2024-08-18T21:19:26.125 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-08-18T21:19:26.165 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-08-18T21:19:26.485 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-08-18T21:19:26.512 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-18T21:19:26.551 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-mgr-cephadm (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:26.847 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-repoze.lru. 2024-08-18T21:19:26.874 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-08-18T21:19:26.905 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-08-18T21:19:27.216 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-routes. 2024-08-18T21:19:27.243 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-08-18T21:19:27.283 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-08-18T21:19:27.686 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-08-18T21:19:27.713 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-18T21:19:27.753 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-mgr-dashboard (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:30.642 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-08-18T21:19:30.659 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-08-18T21:19:30.700 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-08-18T21:19:31.196 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-joblib. 2024-08-18T21:19:31.223 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-08-18T21:19:31.262 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-08-18T21:19:31.607 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-08-18T21:19:31.623 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-08-18T21:19:31.657 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-08-18T21:19:31.952 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-sklearn. 2024-08-18T21:19:31.979 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-08-18T21:19:32.018 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-08-18T21:19:32.694 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-08-18T21:19:32.722 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-18T21:19:32.854 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:33.909 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-cachetools. 2024-08-18T21:19:33.936 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-08-18T21:19:33.975 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-08-18T21:19:34.228 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-pyasn1. 2024-08-18T21:19:34.244 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-08-18T21:19:34.278 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-08-18T21:19:34.598 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-08-18T21:19:34.626 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-08-18T21:19:34.664 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-08-18T21:19:34.935 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-rsa. 2024-08-18T21:19:34.951 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-08-18T21:19:35.018 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-rsa (4.8-1) ... 2024-08-18T21:19:35.338 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-google-auth. 2024-08-18T21:19:35.366 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-08-18T21:19:35.405 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-08-18T21:19:35.733 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-08-18T21:19:35.761 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-08-18T21:19:35.800 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-08-18T21:19:36.069 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-websocket. 2024-08-18T21:19:36.086 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-08-18T21:19:36.128 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-08-18T21:19:36.415 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-kubernetes. 2024-08-18T21:19:36.443 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-08-18T21:19:36.473 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-08-18T21:19:37.162 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-08-18T21:19:37.190 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-18T21:19:37.228 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:37.523 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libonig5:amd64. 2024-08-18T21:19:37.551 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-08-18T21:19:37.598 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-08-18T21:19:38.143 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libjq1:amd64. 2024-08-18T21:19:38.171 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-08-18T21:19:38.210 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-08-18T21:19:38.471 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package jq. 2024-08-18T21:19:38.499 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-08-18T21:19:38.537 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-08-18T21:19:38.748 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package socat. 2024-08-18T21:19:38.764 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-08-18T21:19:38.798 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-08-18T21:19:39.093 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package xmlstarlet. 2024-08-18T21:19:39.118 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-08-18T21:19:39.162 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-08-18T21:19:39.506 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-test. 2024-08-18T21:19:39.522 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../097-ceph-test_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:39.556 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-test (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:42.254 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package ceph-volume. 2024-08-18T21:19:42.282 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-5847-g79766939-1jammy_all.deb ... 2024-08-18T21:19:42.320 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking ceph-volume (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:42.624 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package libcephfs-dev. 2024-08-18T21:19:42.640 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:42.682 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking libcephfs-dev (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:43.011 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package lua-socket:amd64. 2024-08-18T21:19:43.038 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-08-18T21:19:43.077 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-08-18T21:19:43.407 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package lua-sec:amd64. 2024-08-18T21:19:43.423 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-08-18T21:19:43.457 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-08-18T21:19:43.836 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package nvme-cli. 2024-08-18T21:19:43.864 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-08-18T21:19:43.902 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-08-18T21:19:44.325 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package pkg-config. 2024-08-18T21:19:44.353 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-08-18T21:19:44.391 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-08-18T21:19:44.652 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-08-18T21:19:44.668 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-08-18T21:19:44.710 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-08-18T21:19:45.156 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-iniconfig. 2024-08-18T21:19:45.184 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-08-18T21:19:45.222 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-08-18T21:19:45.492 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-pastescript. 2024-08-18T21:19:45.520 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-08-18T21:19:45.549 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-08-18T21:19:45.845 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-pluggy. 2024-08-18T21:19:45.873 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-08-18T21:19:45.911 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-08-18T21:19:46.223 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-psutil. 2024-08-18T21:19:46.251 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-08-18T21:19:46.289 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-08-18T21:19:46.659 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-py. 2024-08-18T21:19:46.688 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-08-18T21:19:46.726 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-py (1.10.0-1) ... 2024-08-18T21:19:47.054 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-pygments. 2024-08-18T21:19:47.083 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-08-18T21:19:47.120 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-08-18T21:19:47.566 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-pyinotify. 2024-08-18T21:19:47.594 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-08-18T21:19:47.633 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-08-18T21:19:47.919 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-toml. 2024-08-18T21:19:47.947 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-08-18T21:19:47.985 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-08-18T21:19:48.255 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-pytest. 2024-08-18T21:19:48.283 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-08-18T21:19:48.321 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-08-18T21:19:48.667 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package python3-simplejson. 2024-08-18T21:19:48.695 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-08-18T21:19:48.733 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-08-18T21:19:49.011 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-08-18T21:19:49.028 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-08-18T21:19:49.069 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-08-18T21:19:49.824 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package radosgw. 2024-08-18T21:19:49.852 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../116-radosgw_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:49.889 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking radosgw (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:50.830 INFO:teuthology.orchestra.run.smithi099.stdout:Selecting previously unselected package rbd-fuse. 2024-08-18T21:19:50.842 INFO:teuthology.orchestra.run.smithi099.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-5847-g79766939-1jammy_amd64.deb ... 2024-08-18T21:19:50.884 INFO:teuthology.orchestra.run.smithi099.stdout:Unpacking rbd-fuse (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:51.181 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-08-18T21:19:51.431 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-08-18T21:19:51.532 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-08-18T21:19:51.771 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-08-18T21:19:52.121 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-08-18T21:19:52.564 INFO:teuthology.orchestra.run.smithi099.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-08-18T21:19:52.651 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up cephadm (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:53.234 INFO:teuthology.orchestra.run.smithi099.stdout:Adding system user cephadm....done 2024-08-18T21:19:53.486 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-08-18T21:19:54.247 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-08-18T21:19:54.585 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-08-18T21:19:54.787 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-08-18T21:19:55.047 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-08-18T21:19:55.298 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-08-18T21:19:55.399 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-py (1.10.0-1) ... 2024-08-18T21:19:55.684 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-08-18T21:19:56.045 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-08-18T21:19:56.304 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-08-18T21:19:56.605 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-08-18T21:19:56.903 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-08-18T21:19:57.250 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-08-18T21:19:57.518 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-08-18T21:19:57.786 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-ceph-argparse (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:19:58.045 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-08-18T21:19:58.171 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-08-18T21:19:58.289 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-08-18T21:19:58.390 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-08-18T21:19:58.632 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-08-18T21:19:59.006 INFO:teuthology.orchestra.run.smithi099.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-08-18T21:19:59.038 INFO:teuthology.orchestra.run.smithi099.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-08-18T21:19:59.094 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-08-18T21:19:59.220 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-08-18T21:19:59.588 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-08-18T21:19:59.697 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-08-18T21:19:59.940 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-08-18T21:20:00.200 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-08-18T21:20:00.501 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up zip (3.0-12build2) ... 2024-08-18T21:20:00.627 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-08-18T21:20:01.337 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-08-18T21:20:01.589 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-08-18T21:20:01.689 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-08-18T21:20:01.790 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-08-18T21:20:02.092 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-08-18T21:20:02.493 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-08-18T21:20:02.752 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-08-18T21:20:03.121 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-08-18T21:20:03.414 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-08-18T21:20:03.740 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-08-18T21:20:03.991 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-08-18T21:20:04.117 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-ceph-common (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:04.427 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-08-18T21:20:04.745 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-08-18T21:20:05.926 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-08-18T21:20:06.169 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-18T21:20:06.306 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-toml (0.10.2-1) ... 2024-08-18T21:20:06.596 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-08-18T21:20:06.697 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-08-18T21:20:06.798 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-08-18T21:20:07.075 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-08-18T21:20:07.334 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-08-18T21:20:07.585 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-18T21:20:07.728 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-08-18T21:20:08.030 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-08-18T21:20:08.264 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-08-18T21:20:08.498 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-08-18T21:20:08.758 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-08-18T21:20:09.017 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-08-18T21:20:09.285 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-08-18T21:20:09.411 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-08-18T21:20:09.721 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-08-18T21:20:09.838 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-08-18T21:20:10.131 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-08-18T21:20:10.416 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-08-18T21:20:10.684 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-08-18T21:20:10.810 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-08-18T21:20:10.936 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-08-18T21:20:11.025 INFO:teuthology.orchestra.run.smithi099.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-08-18T21:20:11.304 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-08-18T21:20:11.405 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-08-18T21:20:11.790 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-08-18T21:20:12.050 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up lua-any (27ubuntu1) ... 2024-08-18T21:20:12.176 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-portend (3.0.0-1) ... 2024-08-18T21:20:12.435 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-08-18T21:20:12.561 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-08-18T21:20:12.887 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-08-18T21:20:13.163 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-08-18T21:20:13.448 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-08-18T21:20:13.583 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-rsa (4.8-1) ... 2024-08-18T21:20:13.859 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-08-18T21:20:14.135 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-08-18T21:20:14.672 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-08-18T21:20:15.040 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-08-18T21:20:15.325 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-08-18T21:20:15.661 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-08-18T21:20:15.787 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up librados2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:15.913 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libsqlite3-mod-ceph (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:16.039 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-08-18T21:20:16.385 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libcephfs2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:16.509 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libradosstriper1 (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:16.644 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-08-18T21:20:16.904 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up librbd1 (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:17.005 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-mgr-modules-core (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:17.106 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-fuse (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:17.317 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-08-18T21:20:17.317 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-08-18T21:20:17.841 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up libcephfs-dev (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:17.967 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-rados (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:18.084 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up librgw2 (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:18.194 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-08-18T21:20:19.389 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-rbd (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:19.515 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up rbd-fuse (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:19.641 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-rgw (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:19.759 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up python3-cephfs (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:19.894 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-common (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:20.445 INFO:teuthology.orchestra.run.smithi099.stdout:Adding group ceph....done 2024-08-18T21:20:20.838 INFO:teuthology.orchestra.run.smithi099.stdout:Adding system user ceph....done 2024-08-18T21:20:20.990 INFO:teuthology.orchestra.run.smithi099.stdout:Setting system user ceph properties....done 2024-08-18T21:20:21.004 INFO:teuthology.orchestra.run.smithi099.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-08-18T21:20:21.142 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-08-18T21:20:21.489 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-08-18T21:20:21.929 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-test (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:22.029 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up radosgw (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:22.470 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-08-18T21:20:22.471 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-08-18T21:20:22.907 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-base (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:23.255 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-08-18T21:20:23.725 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-mds (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:23.960 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-08-18T21:20:23.960 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-08-18T21:20:24.452 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-mgr (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:24.647 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-08-18T21:20:24.647 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-08-18T21:20:25.258 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-osd (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:25.547 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-08-18T21:20:25.547 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-08-18T21:20:26.043 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-mgr-k8sevents (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:26.170 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:26.320 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-mon (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:26.515 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-08-18T21:20:26.515 INFO:teuthology.orchestra.run.smithi099.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-08-18T21:20:26.988 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-mgr-cephadm (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:27.139 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:27.265 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-mgr-dashboard (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:27.410 INFO:teuthology.orchestra.run.smithi099.stdout:Setting up ceph-volume (19.0.0-5847-g79766939-1jammy) ... 2024-08-18T21:20:27.778 INFO:teuthology.orchestra.run.smithi099.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-08-18T21:20:27.895 INFO:teuthology.orchestra.run.smithi099.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-08-18T21:20:28.022 INFO:teuthology.orchestra.run.smithi099.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-08-18T21:20:29.075 INFO:teuthology.orchestra.run.smithi099.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-08-18T21:20:29.756 INFO:teuthology.orchestra.run.smithi099.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-08-18T21:20:29.764 DEBUG:teuthology.parallel:result is None 2024-08-18T21:20:29.765 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=797669399b32d2915e64eb6d4b1257ae3c50f80b 2024-08-18T21:20:29.917 DEBUG:teuthology.orchestra.run.smithi099:> dpkg-query -W -f '${Version}' ceph 2024-08-18T21:20:29.932 INFO:teuthology.orchestra.run.smithi099.stdout:19.0.0-5847-g79766939-1jammy 2024-08-18T21:20:29.933 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-5847-g79766939-1jammy 2024-08-18T21:20:29.933 INFO:teuthology.task.install:The correct ceph version 19.0.0-5847-g79766939-1jammy is installed. 2024-08-18T21:20:29.935 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-18T21:20:29.936 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-08-18T21:20:29.936 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-18T21:20:29.986 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-18T21:20:29.986 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-08-18T21:20:29.986 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/daemon-helper 2024-08-18T21:20:30.041 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-18T21:20:30.099 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-18T21:20:30.099 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-08-18T21:20:30.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-18T21:20:30.151 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-18T21:20:30.203 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-18T21:20:30.204 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-08-18T21:20:30.204 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/usr/bin/stdin-killer 2024-08-18T21:20:30.260 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-18T21:20:30.315 INFO:teuthology.run_tasks:Running task workunit... 2024-08-18T21:20:30.332 INFO:tasks.workunit:Pulling workunits from ref 964adda23da9f7b057cb34fdc0b82aeb7ed65324 2024-08-18T21:20:30.333 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-08-18T21:20:30.333 INFO:tasks.workunit:timeout=3h 2024-08-18T21:20:30.333 INFO:tasks.workunit:cleanup=True 2024-08-18T21:20:30.333 DEBUG:teuthology.orchestra.run.smithi099:> stat -- /home/ubuntu/cephtest/mnt.0 2024-08-18T21:20:30.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T21:20:30.365 INFO:teuthology.orchestra.run.smithi099.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-08-18T21:20:30.365 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-08-18T21:20:30.412 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-08-18T21:20:30.413 DEBUG:teuthology.orchestra.run.smithi099:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-08-18T21:20:30.461 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 964adda23da9f7b057cb34fdc0b82aeb7ed65324 2024-08-18T21:20:30.511 INFO:tasks.workunit.client.0.smithi099.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-08-18T21:21:21.977 INFO:tasks.workunit.client.0.smithi099.stderr:Updating files: 97% (12568/12849) Updating files: 98% (12593/12849) Updating files: 99% (12721/12849) Updating files: 100% (12849/12849) Updating files: 100% (12849/12849), done. 2024-08-18T21:21:22.641 INFO:tasks.workunit.client.0.smithi099.stderr:Note: switching to '964adda23da9f7b057cb34fdc0b82aeb7ed65324'. 2024-08-18T21:21:22.641 INFO:tasks.workunit.client.0.smithi099.stderr: 2024-08-18T21:21:22.641 INFO:tasks.workunit.client.0.smithi099.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr:changes and commit them, and you can discard any commits you make in this 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr:state without impacting any branches by switching back to a branch. 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr: 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr:If you want to create a new branch to retain commits you create, you may 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr:do so (now or later) by using -c with the switch command. Example: 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr: 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr: git switch -c 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr: 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr:Or undo this operation with: 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr: 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr: git switch - 2024-08-18T21:21:22.642 INFO:tasks.workunit.client.0.smithi099.stderr: 2024-08-18T21:21:22.643 INFO:tasks.workunit.client.0.smithi099.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-08-18T21:21:22.643 INFO:tasks.workunit.client.0.smithi099.stderr: 2024-08-18T21:21:22.643 INFO:tasks.workunit.client.0.smithi099.stderr:HEAD is now at 964adda23da Merge pull request #59290 from anthonyeleven/mountpoint 2024-08-18T21:21:22.644 DEBUG:teuthology.orchestra.run.smithi099:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-08-18T21:21:22.693 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-08-18T21:21:22.693 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-08-18T21:21:22.740 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-08-18T21:21:22.740 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-08-18T21:21:22.741 DEBUG:teuthology.orchestra.run.smithi099:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-08-18T21:21:22.801 INFO:tasks.workunit.client.0.smithi099.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-18T21:21:22.832 INFO:tasks.workunit.client.0.smithi099.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-18T21:21:22.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-18T21:21:22.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-18T21:21:22.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export PYTHONWARNINGS=ignore 2024-08-18T21:21:22.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: PYTHONWARNINGS=ignore 2024-08-18T21:21:22.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: export CEPH_CONF=/dev/null 2024-08-18T21:21:22.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: CEPH_CONF=/dev/null 2024-08-18T21:21:22.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: unset CEPH_ARGS 2024-08-18T21:21:22.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: local code 2024-08-18T21:21:22.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: run td/bad-inc-map 2024-08-18T21:21:22.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-08-18T21:21:22.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-08-18T21:21:22.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:58219 2024-08-18T21:21:22.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:58219 2024-08-18T21:21:22.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-08-18T21:21:22.833 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-08-18T21:21:22.834 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=0ee7395d-d02a-44c2-bfc3-4337edf6c900 --auth-supported=none ' 2024-08-18T21:21:22.834 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:58219 ' 2024-08-18T21:21:22.834 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-08-18T21:21:22.834 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-08-18T21:21:22.834 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-18T21:21:22.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-08-18T21:21:22.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-08-18T21:21:22.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-08-18T21:21:22.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2024-08-18T21:21:22.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2024-08-18T21:21:22.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-08-18T21:21:22.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:21:22.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-08-18T21:21:22.836 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:21:22.836 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:21:22.837 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:21:22.837 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:21:22.837 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:21:22.837 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:21:22.837 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:21:22.838 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:21:22.838 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:21:22.839 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:21:22.839 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:21:22.839 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:21:22.840 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:21:22.840 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:21:22.840 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:21:22.840 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:21:22.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:21:22.842 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:21:22.842 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-08-18T21:21:22.843 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:21:22.843 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:21:22.843 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21446 2024-08-18T21:21:22.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21446 2024-08-18T21:21:22.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:21:22.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:21:22.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2024-08-18T21:21:22.845 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-18T21:21:22.845 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:21:22.845 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21446 2024-08-18T21:21:22.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21446 2024-08-18T21:21:22.848 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-18T21:21:22.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-18T21:21:22.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-18T21:21:22.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-18T21:21:22.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-08-18T21:21:22.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-08-18T21:21:22.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-08-18T21:21:22.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-08-18T21:21:22.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2024-08-18T21:21:22.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-18T21:21:22.849 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-18T21:21:22.849 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-18T21:21:22.849 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2024-08-18T21:21:22.849 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-08-18T21:21:23.450 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-18T21:21:23.450 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:21:23.450 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:21:23.450 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:21:23.450 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:21:23.450 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21446 2024-08-18T21:21:23.450 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21446/$cluster-$name.asok' 2024-08-18T21:21:23.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21446/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-18T21:21:23.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-18T21:21:23.690 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-18T21:21:23.690 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:21:23.690 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:21:23.690 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-18T21:21:23.693 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-18T21:21:23.693 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:21:23.693 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:21:23.694 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:21:23.694 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:21:23.694 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:21:23.694 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21446 2024-08-18T21:21:23.694 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21446/ceph-mon.a.asok 2024-08-18T21:21:23.694 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:21:23.694 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21446/ceph-mon.a.asok config get fsid 2024-08-18T21:21:23.822 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-18T21:21:23.822 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:21:23.822 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:21:23.822 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-18T21:21:23.822 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-18T21:21:23.822 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:21:23.822 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:21:23.822 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:21:23.822 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:21:23.823 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:21:23.823 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21446 2024-08-18T21:21:23.823 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21446/ceph-mon.a.asok 2024-08-18T21:21:23.823 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:21:23.823 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21446/ceph-mon.a.asok config get mon_host 2024-08-18T21:21:23.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-08-18T21:21:23.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2024-08-18T21:21:23.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-18T21:21:23.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-18T21:21:23.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-18T21:21:23.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2024-08-18T21:21:23.924 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-18T21:21:24.819 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-18T21:21:24.819 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:21:24.819 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:21:24.820 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:21:24.820 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:21:24.820 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21446 2024-08-18T21:21:24.820 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21446/$cluster-$name.asok' 2024-08-18T21:21:24.820 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:21:24.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.21446/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:21:24.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ee7395d-d02a-44c2-bfc3-4337edf6c900 --auth-supported=none --mon-host=127.0.0.1:58219 ' 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-08-18T21:21:24.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21446 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21446/$cluster-$name.asok' 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21446/$cluster-$name.asok' 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:21:24.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:21:24.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:21:24.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-08-18T21:21:24.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-08-18T21:21:24.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:21:24.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:21:24.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:21:24.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:21:24.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:21:24.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:21:24.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2024-08-18T21:21:24.892 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:21:24.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a06c5acc-5c9e-476c-a3bd-12970fc96c0c 2024-08-18T21:21:24.892 INFO:tasks.workunit.client.0.smithi099.stdout:add osd0 a06c5acc-5c9e-476c-a3bd-12970fc96c0c 2024-08-18T21:21:24.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a06c5acc-5c9e-476c-a3bd-12970fc96c0c' 2024-08-18T21:21:24.893 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:21:24.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUZcJmjLodNhAAJHWFxQOaYbx0cnvZrJjp3w== 2024-08-18T21:21:24.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUZcJmjLodNhAAJHWFxQOaYbx0cnvZrJjp3w=="}' 2024-08-18T21:21:24.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a06c5acc-5c9e-476c-a3bd-12970fc96c0c -i td/bad-inc-map/0/new.json 2024-08-18T21:21:25.211 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:21:25.227 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2024-08-18T21:21:25.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=0ee7395d-d02a-44c2-bfc3-4337edf6c900 --auth-supported=none --mon-host=127.0.0.1:58219 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21446/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUZcJmjLodNhAAJHWFxQOaYbx0cnvZrJjp3w== --osd-uuid a06c5acc-5c9e-476c-a3bd-12970fc96c0c 2024-08-18T21:21:25.308 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:21:25.306+0000 7fc79607f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:21:25.311 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:21:25.306+0000 7fc79607f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:21:25.312 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:21:25.310+0000 7fc79607f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:21:25.313 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:21:25.310+0000 7fc79607f7c0 -1 bdev(0x56249d2f1c00 td/bad-inc-map/0/block) open stat got: (1) Operation not permitted 2024-08-18T21:21:25.313 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:21:25.310+0000 7fc79607f7c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-08-18T21:21:32.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-08-18T21:21:32.041 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:21:32.042 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd0 key to auth repository 2024-08-18T21:21:32.042 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-18T21:21:32.042 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:21:32.287 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:21:32.288 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-18T21:21:32.288 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:21:32.288 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=0ee7395d-d02a-44c2-bfc3-4337edf6c900 --auth-supported=none --mon-host=127.0.0.1:58219 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21446/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:21:32.288 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:21:32.288 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:21:32.317 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:21:32.314+0000 7ff6a91b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:21:32.320 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:21:32.318+0000 7ff6a91b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:21:32.321 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:21:32.318+0000 7ff6a91b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:21:32.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-18T21:21:32.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:21:32.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:21:32.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:21:32.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:21:32.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:32.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:21:32.509 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:21:32.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:32.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:32.735 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:33.736 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:33.736 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:33.737 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:21:33.737 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:21:33.737 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:33.737 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:34.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:34.162 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:21:34.157+0000 7ff6a91b77c0 -1 Falling back to public interface 2024-08-18T21:21:35.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:35.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:35.046 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:21:35.047 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:21:35.047 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:35.047 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:35.460 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:36.462 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:36.462 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:36.463 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:21:36.463 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:21:36.463 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:36.463 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:36.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:37.017 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:21:37.013+0000 7ff6a91b77c0 -1 osd.0 0 log_to_monitors true 2024-08-18T21:21:37.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:37.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:37.892 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:21:37.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:21:37.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:37.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:38.328 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:39.330 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:39.330 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:39.330 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:21:39.331 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:21:39.331 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:39.331 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:39.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:40.761 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:40.761 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:40.762 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:21:40.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:21:40.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:40.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:41.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:42.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:42.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:42.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:21:42.208 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:21:42.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:42.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:42.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:43.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:43.663 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:21:43.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:43.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:21:43.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:43.664 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:44.210 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:45.212 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:45.212 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:45.212 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:21:45.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:21:45.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:45.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:45.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:46.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:46.674 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:46.674 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:21:46.674 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:21:46.674 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:46.675 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:47.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:48.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:48.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:48.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:21:48.199 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:21:48.199 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:48.199 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:48.649 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:49.650 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:49.651 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:49.651 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:21:49.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:21:49.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:49.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:50.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:51.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:51.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:51.191 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:21:51.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:21:51.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:51.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:51.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:52.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:52.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:52.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:21:52.825 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:21:52.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:52.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:53.488 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:54.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:54.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:54.490 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:21:54.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:21:54.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:54.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:55.435 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:56.436 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:56.436 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:56.436 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:21:56.437 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:21:56.437 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:56.437 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:57.100 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:58.102 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:58.102 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:58.102 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:21:58.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:21:58.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:58.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:21:58.640 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:21:59.642 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:21:59.642 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:21:59.642 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:21:59.642 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:21:59.643 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:21:59.643 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:22:00.470 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:01.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:01.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:01.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-18T21:22:01.473 INFO:tasks.workunit.client.0.smithi099.stdout:19 2024-08-18T21:22:01.473 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:01.473 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:22:02.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:03.042 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:03.042 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:03.042 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-08-18T21:22:03.042 INFO:tasks.workunit.client.0.smithi099.stdout:20 2024-08-18T21:22:03.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:03.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:22:03.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:04.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:04.869 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:04.869 INFO:tasks.workunit.client.0.smithi099.stdout:21 2024-08-18T21:22:04.869 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-08-18T21:22:04.869 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:04.870 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:22:05.338 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1263384529,v1:127.0.0.1:6803/1263384529] [v2:127.0.0.1:6804/1263384529,v1:127.0.0.1:6805/1263384529] exists,up a06c5acc-5c9e-476c-a3bd-12970fc96c0c 2024-08-18T21:22:05.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:22:05.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:22:05.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:22:05.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-08-18T21:22:05.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-08-18T21:22:05.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:22:05.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-18T21:22:05.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:22:05.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2024-08-18T21:22:05.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ee7395d-d02a-44c2-bfc3-4337edf6c900 --auth-supported=none --mon-host=127.0.0.1:58219 ' 2024-08-18T21:22:05.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:22:05.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:22:05.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:22:05.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-08-18T21:22:05.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-08-18T21:22:05.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:22:05.343 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:22:05.343 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-08-18T21:22:05.343 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:22:05.343 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:22:05.343 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:22:05.343 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21446 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21446/$cluster-$name.asok' 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21446/$cluster-$name.asok' 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:22:05.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:22:05.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:22:05.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:22:05.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2024-08-18T21:22:05.345 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:22:05.345 INFO:tasks.workunit.client.0.smithi099.stdout:add osd1 5482872c-9890-41e3-8f6f-01c253c98699 2024-08-18T21:22:05.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5482872c-9890-41e3-8f6f-01c253c98699 2024-08-18T21:22:05.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5482872c-9890-41e3-8f6f-01c253c98699' 2024-08-18T21:22:05.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:22:05.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9ZcJmWiW7FRAAWEORfnrnFG1196XD4SCT/Q== 2024-08-18T21:22:05.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9ZcJmWiW7FRAAWEORfnrnFG1196XD4SCT/Q=="}' 2024-08-18T21:22:05.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5482872c-9890-41e3-8f6f-01c253c98699 -i td/bad-inc-map/1/new.json 2024-08-18T21:22:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:22:05.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2024-08-18T21:22:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=0ee7395d-d02a-44c2-bfc3-4337edf6c900 --auth-supported=none --mon-host=127.0.0.1:58219 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21446/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9ZcJmWiW7FRAAWEORfnrnFG1196XD4SCT/Q== --osd-uuid 5482872c-9890-41e3-8f6f-01c253c98699 2024-08-18T21:22:05.915 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:05.917+0000 7f4b14d7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:05.923 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:05.925+0000 7f4b14d7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:05.925 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:05.925+0000 7f4b14d7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:05.925 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:05.925+0000 7f4b14d7c7c0 -1 bdev(0x561008e7fc00 td/bad-inc-map/1/block) open stat got: (1) Operation not permitted 2024-08-18T21:22:05.925 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:05.925+0000 7f4b14d7c7c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-08-18T21:22:13.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-08-18T21:22:13.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:22:13.002 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd1 key to auth repository 2024-08-18T21:22:13.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-18T21:22:13.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:22:13.577 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:22:13.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-18T21:22:13.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:22:13.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=0ee7395d-d02a-44c2-bfc3-4337edf6c900 --auth-supported=none --mon-host=127.0.0.1:58219 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21446/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:22:13.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:22:13.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:22:13.609 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:13.608+0000 7fbbbdf417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:13.611 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:13.612+0000 7fbbbdf417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:13.612 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:13.612+0000 7fbbbdf417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:14.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-18T21:22:14.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:22:14.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:22:14.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:22:14.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:22:14.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:14.035 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:22:14.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:22:14.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:14.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:14.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:14.882 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:14.880+0000 7fbbbdf417c0 -1 Falling back to public interface 2024-08-18T21:22:15.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:15.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:15.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:22:15.491 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:22:15.492 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:15.492 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:15.980 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:16.982 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:16.982 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:16.982 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:22:16.982 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:22:16.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:16.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:17.405 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:17.833 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:17.832+0000 7fbbbdf417c0 -1 osd.1 0 log_to_monitors true 2024-08-18T21:22:18.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:18.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:18.408 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:22:18.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:22:18.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:18.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:19.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:20.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:20.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:20.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:22:20.011 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:22:20.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:20.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:20.542 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:21.544 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:21.544 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:21.544 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:22:21.544 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:22:21.545 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:21.545 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:22.036 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:23.038 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:23.038 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:23.038 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:22:23.039 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:22:23.039 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:23.039 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:23.470 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:24.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:24.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:24.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:22:24.472 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:22:24.473 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:24.473 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:25.025 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:26.027 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:26.027 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:26.027 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:22:26.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:22:26.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:26.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:26.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:27.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:27.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:27.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:22:27.508 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:22:27.508 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:27.508 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:28.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:29.030 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:29.030 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:29.030 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:22:29.031 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:22:29.031 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:29.031 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:29.475 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:30.476 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:30.476 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:30.476 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:22:30.477 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:22:30.477 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:30.477 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:30.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:31.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:31.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:22:31.972 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:22:31.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:31.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:32.949 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:33.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:33.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:33.951 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:22:33.952 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:22:33.952 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:33.952 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:35.233 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:36.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:36.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:36.235 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:22:36.236 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:22:36.236 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:36.236 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:37.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:38.175 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:38.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:38.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:22:38.176 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:22:38.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:38.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:38.995 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:39.996 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:39.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:39.997 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:22:39.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:22:39.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:39.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:41.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:42.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:42.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:42.258 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:22:42.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:22:42.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:42.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:43.207 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:44.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:44.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:44.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:22:44.209 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:22:44.210 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:44.210 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:22:44.682 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/118125125,v1:127.0.0.1:6811/118125125] [v2:127.0.0.1:6812/118125125,v1:127.0.0.1:6813/118125125] exists,up 5482872c-9890-41e3-8f6f-01c253c98699 2024-08-18T21:22:44.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:22:44.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:22:44.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:22:44.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-08-18T21:22:44.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=0ee7395d-d02a-44c2-bfc3-4337edf6c900 --auth-supported=none --mon-host=127.0.0.1:58219 ' 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-08-18T21:22:44.685 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:22:44.686 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:22:44.686 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:22:44.687 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:22:44.687 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:22:44.687 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21446 2024-08-18T21:22:44.687 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21446/$cluster-$name.asok' 2024-08-18T21:22:44.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21446/$cluster-$name.asok' 2024-08-18T21:22:44.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:22:44.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:22:44.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:22:44.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-08-18T21:22:44.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-08-18T21:22:44.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:22:44.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:22:44.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:22:44.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:22:44.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:22:44.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:22:44.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2024-08-18T21:22:44.689 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:22:44.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cb8a4ee4-afab-4314-9f47-a90d58cb6645 2024-08-18T21:22:44.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cb8a4ee4-afab-4314-9f47-a90d58cb6645' 2024-08-18T21:22:44.690 INFO:tasks.workunit.client.0.smithi099.stdout:add osd2 cb8a4ee4-afab-4314-9f47-a90d58cb6645 2024-08-18T21:22:44.690 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:22:44.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAkZsJmOVNJKhAA6VPPrvcxg5F/Avx2OWH85Q== 2024-08-18T21:22:44.710 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAkZsJmOVNJKhAA6VPPrvcxg5F/Avx2OWH85Q=="}' 2024-08-18T21:22:44.710 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cb8a4ee4-afab-4314-9f47-a90d58cb6645 -i td/bad-inc-map/2/new.json 2024-08-18T21:22:45.194 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:22:45.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2024-08-18T21:22:45.210 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=0ee7395d-d02a-44c2-bfc3-4337edf6c900 --auth-supported=none --mon-host=127.0.0.1:58219 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21446/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkZsJmOVNJKhAA6VPPrvcxg5F/Avx2OWH85Q== --osd-uuid cb8a4ee4-afab-4314-9f47-a90d58cb6645 2024-08-18T21:22:45.243 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:45.239+0000 7f79207607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:45.311 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:45.307+0000 7f79207607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:45.313 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:45.311+0000 7f79207607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:45.313 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:45.311+0000 7f79207607c0 -1 bdev(0x55d81307c800 td/bad-inc-map/2/block) open stat got: (1) Operation not permitted 2024-08-18T21:22:45.313 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:45.311+0000 7f79207607c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-08-18T21:22:51.937 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-08-18T21:22:51.938 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:22:51.938 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd2 key to auth repository 2024-08-18T21:22:51.939 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-18T21:22:51.939 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:22:52.533 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:22:52.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-18T21:22:52.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:22:52.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=0ee7395d-d02a-44c2-bfc3-4337edf6c900 --auth-supported=none --mon-host=127.0.0.1:58219 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.21446/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:22:52.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:22:52.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:22:52.564 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:52.559+0000 7ffbc46db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:52.587 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:52.583+0000 7ffbc46db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:52.588 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:52.587+0000 7ffbc46db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:22:52.963 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-18T21:22:52.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:22:52.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:22:52.964 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:22:52.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:22:52.965 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:22:52.965 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:52.965 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:22:52.965 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:52.965 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:22:53.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:53.851 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:53.847+0000 7ffbc46db7c0 -1 Falling back to public interface 2024-08-18T21:22:54.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:54.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:54.376 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:22:54.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:22:54.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:54.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:22:54.786 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:55.788 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:55.788 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:55.788 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:22:55.789 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:22:55.789 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:55.789 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:22:56.286 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:56.871 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:22:56.866+0000 7ffbc46db7c0 -1 osd.2 0 log_to_monitors true 2024-08-18T21:22:57.290 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:57.291 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:57.291 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:22:57.291 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:22:57.291 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:57.291 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:22:57.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:22:58.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:22:58.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:22:58.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:22:58.808 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:22:58.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:22:58.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:22:59.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:00.280 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:00.280 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:00.280 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:23:00.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:23:00.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:00.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:00.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:01.761 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:01.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:01.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:23:01.762 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:23:01.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:01.763 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:02.210 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:03.212 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:03.212 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:03.212 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:23:03.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:23:03.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:03.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:03.793 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:04.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:04.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:04.795 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:23:04.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:23:04.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:04.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:05.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:06.276 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:06.276 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:06.276 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:23:06.276 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:23:06.277 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:06.277 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:06.755 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:07.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:07.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:07.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:23:07.757 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:23:07.758 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:07.758 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:08.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:09.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:09.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:09.191 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:23:09.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:23:09.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:09.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:09.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:10.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:10.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:10.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:23:10.806 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:23:10.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:10.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:11.287 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:12.289 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:12.289 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:12.289 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:23:12.290 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:23:12.290 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:12.290 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:13.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:14.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:14.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:14.535 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:23:14.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:23:14.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:14.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:15.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:16.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:16.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:16.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:23:16.369 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:23:16.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:16.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:17.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:18.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:18.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:18.274 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:23:18.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:23:18.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:18.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:19.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:20.199 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:20.199 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:20.200 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:23:20.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:23:20.201 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:20.201 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:21.635 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:22.637 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:22.637 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:22.637 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:23:22.637 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:23:22.638 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:22.638 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:23.090 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2032858830,v1:127.0.0.1:6819/2032858830] [v2:127.0.0.1:6820/2032858830,v1:127.0.0.1:6821/2032858830] exists,up cb8a4ee4-afab-4314-9f47-a90d58cb6645 2024-08-18T21:23:23.090 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:23:23.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:23:23.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:23:23.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-08-18T21:23:23.581 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-08-18T21:23:23.581 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2024-08-18T21:23:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'foo' already exists 2024-08-18T21:23:24.390 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-18T21:23:25.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-08-18T21:23:27.349 INFO:tasks.workunit.client.0.smithi099.stderr:set pool 1 min_size to 1 2024-08-18T21:23:27.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-08-18T21:23:28.806 INFO:tasks.workunit.client.0.smithi099.stderr:set pool 1 min_size to 2 2024-08-18T21:23:28.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-08-18T21:23:33.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-08-18T21:23:33.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-08-18T21:23:33.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:23:33.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:23:33.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:23:33.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:23:33.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-08-18T21:23:33.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:23:33.836 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:23:33.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:33.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:23:34.300 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1263384529,v1:127.0.0.1:6803/1263384529] [v2:127.0.0.1:6804/1263384529,v1:127.0.0.1:6805/1263384529] exists,up a06c5acc-5c9e-476c-a3bd-12970fc96c0c 2024-08-18T21:23:34.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:23:34.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:23:34.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:23:34.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-08-18T21:23:34.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-08-18T21:23:34.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:23:34.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:23:34.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:23:34.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:23:34.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-08-18T21:23:34.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:23:34.303 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:23:34.303 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:34.303 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:23:34.714 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 10 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/118125125,v1:127.0.0.1:6811/118125125] [v2:127.0.0.1:6812/118125125,v1:127.0.0.1:6813/118125125] exists,up 5482872c-9890-41e3-8f6f-01c253c98699 2024-08-18T21:23:34.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:23:34.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:23:34.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:23:34.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-08-18T21:23:34.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-08-18T21:23:34.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:23:34.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:23:34.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:23:34.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:23:34.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-08-18T21:23:34.717 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:23:34.717 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:23:34.717 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:34.717 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:23:35.158 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 15 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2032858830,v1:127.0.0.1:6819/2032858830] [v2:127.0.0.1:6820/2032858830,v1:127.0.0.1:6821/2032858830] exists,up cb8a4ee4-afab-4314-9f47-a90d58cb6645 2024-08-18T21:23:35.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:23:35.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:23:35.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:23:35.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-08-18T21:23:35.160 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:24.109+0000 7ffbb567f640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.160 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:24.365+0000 7ffbb567f640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.160 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:25.325+0000 7ffbb567f640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.160 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:26.237+0000 7ffbb367b640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.160 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:26.517+0000 7ffbb567f640 -1 osd.2 20 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.161 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:26.693+0000 7ffbb367b640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.161 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:27.053+0000 7ffbb567f640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.161 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:27.369+0000 7ffbb567f640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.161 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:28.137+0000 7ffbb567f640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.161 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:28.825+0000 7ffbb567f640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.161 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:28.829+0000 7ffbb567f640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.161 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:29.737+0000 7ffbb567f640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-08-18T21:23:35.161 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-08-18T21:23:35.163 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:24.109+0000 7ffbb567f640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-08-18T21:23:35.164 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:24.365+0000 7ffbb567f640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-08-18T21:23:35.164 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:25.325+0000 7ffbb567f640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-08-18T21:23:35.164 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:26.237+0000 7ffbb367b640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-08-18T21:23:35.164 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:26.517+0000 7ffbb567f640 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2024-08-18T21:23:35.164 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:26.693+0000 7ffbb367b640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-08-18T21:23:35.164 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:27.053+0000 7ffbb567f640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-08-18T21:23:35.164 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:27.369+0000 7ffbb567f640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-08-18T21:23:35.164 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:28.141+0000 7ffbb567f640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-08-18T21:23:35.164 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:28.825+0000 7ffbb567f640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-08-18T21:23:35.164 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:28.829+0000 7ffbb567f640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-08-18T21:23:35.164 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:23:29.737+0000 7ffbb567f640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-08-18T21:23:35.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-08-18T21:23:35.165 INFO:tasks.workunit.client.0.smithi099.stdout:success 2024-08-18T21:23:35.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-08-18T21:23:35.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2024-08-18T21:23:35.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-08-18T21:23:35.699 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'foo' does not exist 2024-08-18T21:23:35.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-08-18T21:23:35.716 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:23:35.716 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:23:35.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:23:35.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:23:35.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:23:41.032 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:23:41.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-08-18T21:23:41.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-08-18T21:23:41.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:23:41.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-08-18T21:23:41.034 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:23:41.034 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:23:41.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:23:41.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:23:41.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:23:41.041 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:23:41.041 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:23:41.042 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:23:41.042 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:23:41.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:23:41.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:23:41.043 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:23:41.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:23:41.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:23:41.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:23:41.045 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:23:41.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:23:41.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:23:41.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-08-18T21:23:41.062 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:23:41.062 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:23:41.062 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21446 2024-08-18T21:23:41.062 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21446 2024-08-18T21:23:41.063 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:23:41.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:23:41.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: main: code=0 2024-08-18T21:23:41.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: teardown td/bad-inc-map 0 2024-08-18T21:23:41.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-08-18T21:23:41.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-18T21:23:41.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-08-18T21:23:41.064 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:23:41.064 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:23:41.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:23:41.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:23:41.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:23:41.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:23:41.066 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:23:41.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:23:41.067 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:23:41.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:23:41.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:23:41.068 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:23:41.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:23:41.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:23:41.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:23:41.069 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:23:41.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:23:41.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-18T21:23:41.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-08-18T21:23:41.072 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:23:41.072 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:23:41.072 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21446 2024-08-18T21:23:41.072 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21446 2024-08-18T21:23:41.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:23:41.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:23:41.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: main: return 0 2024-08-18T21:23:41.074 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-18T21:23:41.074 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-18T21:23:41.127 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-08-18T21:23:41.127 DEBUG:teuthology.orchestra.run.smithi099:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-08-18T21:23:41.179 INFO:tasks.workunit.client.0.smithi099.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-18T21:23:41.186 INFO:tasks.workunit.client.0.smithi099.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-18T21:23:41.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-18T21:23:41.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-18T21:23:41.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export PYTHONWARNINGS=ignore 2024-08-18T21:23:41.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: PYTHONWARNINGS=ignore 2024-08-18T21:23:41.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: export CEPH_CONF=/dev/null 2024-08-18T21:23:41.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: CEPH_CONF=/dev/null 2024-08-18T21:23:41.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: unset CEPH_ARGS 2024-08-18T21:23:41.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: local code 2024-08-18T21:23:41.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: run td/divergent-priors 2024-08-18T21:23:41.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-08-18T21:23:41.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-08-18T21:23:41.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-08-18T21:23:41.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-08-18T21:23:41.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-08-18T21:23:41.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-08-18T21:23:41.187 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-08-18T21:23:41.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-08-18T21:23:41.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-08-18T21:23:41.188 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-08-18T21:23:41.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-08-18T21:23:41.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-08-18T21:23:41.188 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-08-18T21:23:41.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-08-18T21:23:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-08-18T21:23:41.189 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-08-18T21:23:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-08-18T21:23:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-08-18T21:23:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-08-18T21:23:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-08-18T21:23:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-08-18T21:23:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-08-18T21:23:41.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-08-18T21:23:41.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-08-18T21:23:41.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-08-18T21:23:41.191 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-08-18T21:23:41.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none ' 2024-08-18T21:23:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-08-18T21:23:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-08-18T21:23:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-08-18T21:23:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:23:41.192 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-08-18T21:23:41.192 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-18T21:23:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-08-18T21:23:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:TEST_divergent_2 2024-08-18T21:23:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:TEST_divergent_3 2024-08-18T21:23:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:TEST_divergent_ec' 2024-08-18T21:23:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-08-18T21:23:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-08-18T21:23:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-08-18T21:23:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-08-18T21:23:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-18T21:23:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:23:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-18T21:23:41.196 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:23:41.196 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:23:41.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:23:41.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:23:41.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:23:41.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:23:41.197 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:23:41.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:23:41.198 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:23:41.199 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:23:41.199 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:23:41.199 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:23:41.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:23:41.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:23:41.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:23:41.200 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:23:41.201 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:23:41.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:23:41.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-18T21:23:41.203 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:23:41.204 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:23:41.204 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:23:41.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27423 2024-08-18T21:23:41.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:23:41.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:23:41.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-08-18T21:23:41.206 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-18T21:23:41.206 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:23:41.206 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:23:41.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27423 2024-08-18T21:23:41.207 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-18T21:23:41.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-18T21:23:41.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-18T21:23:41.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-18T21:23:41.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-08-18T21:23:41.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-08-18T21:23:41.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-08-18T21:23:41.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-08-18T21:23:41.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-08-18T21:23:41.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-08-18T21:23:41.209 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-08-18T21:23:41.209 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-08-18T21:23:41.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-08-18T21:23:41.229 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:23:41.229 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:23:41.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-08-18T21:23:41.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-08-18T21:23:41.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-18T21:23:41.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-18T21:23:41.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-18T21:23:41.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-08-18T21:23:41.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-08-18T21:23:41.724 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-18T21:23:41.724 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:23:41.724 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:23:41.725 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:23:41.725 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:23:41.725 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:23:41.725 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:23:41.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-18T21:23:41.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-18T21:23:41.944 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-18T21:23:41.944 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:23:41.944 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:23:41.945 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-18T21:23:41.945 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-18T21:23:41.945 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:23:41.945 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:23:41.945 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:23:41.946 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:23:41.946 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:23:41.946 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:23:41.946 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-mon.a.asok 2024-08-18T21:23:41.946 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:23:41.946 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27423/ceph-mon.a.asok config get fsid 2024-08-18T21:23:42.064 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-18T21:23:42.064 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:23:42.064 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:23:42.065 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-18T21:23:42.066 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:23:42.066 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-18T21:23:42.066 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:23:42.066 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:23:42.066 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:23:42.066 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:23:42.066 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:23:42.066 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-mon.a.asok 2024-08-18T21:23:42.066 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:23:42.066 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27423/ceph-mon.a.asok config get mon_host 2024-08-18T21:23:42.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-08-18T21:23:42.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-08-18T21:23:42.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-18T21:23:42.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-18T21:23:42.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-18T21:23:42.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-08-18T21:23:42.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-18T21:23:43.114 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-18T21:23:43.114 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:23:43.114 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:23:43.115 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:23:43.115 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:23:43.115 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:23:43.115 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:23:43.115 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:23:43.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:23:43.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-18T21:23:43.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:23:43.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:23:43.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:23:43.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:23:43.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:23:43.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:23:43.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:23:43.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:23:43.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:23:43.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:23:43.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:23:43.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-08-18T21:23:43.158 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:23:43.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=81ca0476-b29b-4284-91e8-459871ca9868 2024-08-18T21:23:43.158 INFO:tasks.workunit.client.0.smithi099.stdout:add osd0 81ca0476-b29b-4284-91e8-459871ca9868 2024-08-18T21:23:43.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 81ca0476-b29b-4284-91e8-459871ca9868' 2024-08-18T21:23:43.159 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:23:43.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfZsJmTo2wChAAzHFg6fmnIHb3mTJn4skhVw== 2024-08-18T21:23:43.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfZsJmTo2wChAAzHFg6fmnIHb3mTJn4skhVw=="}' 2024-08-18T21:23:43.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 81ca0476-b29b-4284-91e8-459871ca9868 -i td/divergent-priors/0/new.json 2024-08-18T21:23:43.504 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:23:43.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-08-18T21:23:43.520 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfZsJmTo2wChAAzHFg6fmnIHb3mTJn4skhVw== --osd-uuid 81ca0476-b29b-4284-91e8-459871ca9868 2024-08-18T21:23:43.562 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:23:43.561+0000 7f5ee24d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:23:43.581 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:23:43.581+0000 7f5ee24d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:23:43.582 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:23:43.581+0000 7f5ee24d17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:23:43.583 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:23:43.581+0000 7f5ee24d17c0 -1 bdev(0x560331346800 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-08-18T21:23:43.583 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:23:43.581+0000 7f5ee24d17c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-08-18T21:23:50.238 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-08-18T21:23:50.238 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:23:50.239 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd0 key to auth repository 2024-08-18T21:23:50.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-18T21:23:50.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:23:50.811 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:23:50.812 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-18T21:23:50.812 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:23:50.812 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:23:50.813 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:23:50.813 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:23:50.852 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:23:50.849+0000 7f97ba4527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:23:50.878 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:23:50.877+0000 7f97ba4527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:23:50.880 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:23:50.877+0000 7f97ba4527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:23:51.266 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-18T21:23:51.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:23:51.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:23:51.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:23:51.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:23:51.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:51.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:23:51.267 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:23:51.268 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:51.268 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:23:51.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:51.899 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:23:51.897+0000 7f97ba4527c0 -1 Falling back to public interface 2024-08-18T21:23:52.761 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:52.762 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:23:52.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:52.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:23:52.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:52.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:23:53.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:54.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:54.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:54.198 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:23:54.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:23:54.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:54.199 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:23:54.614 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:23:54.612+0000 7f97ba4527c0 -1 osd.0 0 log_to_monitors true 2024-08-18T21:23:54.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:55.657 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:55.657 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:55.657 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:23:55.658 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:23:55.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:55.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:23:56.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:57.138 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:57.138 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:57.138 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:23:57.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:23:57.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:57.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:23:57.675 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:23:58.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:23:58.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:23:58.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:23:58.677 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:23:58.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:23:58.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:23:59.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:00.142 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:00.142 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:00.142 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:24:00.143 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:24:00.143 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:00.143 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:00.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:01.686 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:01.686 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:01.686 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:24:01.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:24:01.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:01.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:02.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:03.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:03.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:03.142 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:24:03.142 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:24:03.142 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:03.142 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:03.726 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:04.728 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:04.938 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:04.939 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:24:04.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:24:04.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:04.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:05.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:06.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:06.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:06.189 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:24:06.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:24:06.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:06.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:06.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:07.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:07.712 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:07.712 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:24:07.712 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:24:07.713 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:07.713 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:08.817 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:09.822 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:09.822 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:09.822 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:24:09.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:24:09.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:09.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:10.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:11.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:11.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:11.689 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:24:11.690 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:24:11.690 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:11.690 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:12.571 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:13.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:13.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:13.573 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:24:13.574 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:24:13.574 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:13.574 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:14.400 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:15.401 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:15.402 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:15.402 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:24:15.402 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:24:15.403 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:15.403 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:16.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:17.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:17.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:17.310 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:24:17.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:24:17.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:17.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:18.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:19.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:19.223 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:24:19.224 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:19.224 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:24:19.224 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:19.224 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:24:19.721 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/727840876,v1:127.0.0.1:6803/727840876] [v2:127.0.0.1:6804/727840876,v1:127.0.0.1:6805/727840876] exists,up 81ca0476-b29b-4284-91e8-459871ca9868 2024-08-18T21:24:19.722 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:24:19.722 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:24:19.722 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:24:19.722 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-18T21:24:19.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-18T21:24:19.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:24:19.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:24:19.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:24:19.724 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:24:19.724 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:24:19.724 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:24:19.726 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:24:19.726 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:24:19.726 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:24:19.726 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:24:19.726 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:24:19.726 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:24:19.726 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:24:19.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:24:19.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:24:19.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:24:19.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:24:19.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:24:19.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:24:19.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:24:19.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:24:19.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:24:19.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-08-18T21:24:19.727 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:24:19.728 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=57cc8ad2-e869-467a-9640-a317277ee238 2024-08-18T21:24:19.728 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 57cc8ad2-e869-467a-9640-a317277ee238' 2024-08-18T21:24:19.728 INFO:tasks.workunit.client.0.smithi099.stdout:add osd1 57cc8ad2-e869-467a-9640-a317277ee238 2024-08-18T21:24:19.728 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:24:19.748 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDZsJmAySULBAAQJjskUxSg7VR+2jkm8cWCw== 2024-08-18T21:24:19.748 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDZsJmAySULBAAQJjskUxSg7VR+2jkm8cWCw=="}' 2024-08-18T21:24:19.748 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 57cc8ad2-e869-467a-9640-a317277ee238 -i td/divergent-priors/1/new.json 2024-08-18T21:24:20.300 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:24:20.315 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-08-18T21:24:20.317 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCDZsJmAySULBAAQJjskUxSg7VR+2jkm8cWCw== --osd-uuid 57cc8ad2-e869-467a-9640-a317277ee238 2024-08-18T21:24:20.351 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:20.348+0000 7f297bdd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:24:20.385 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:20.384+0000 7f297bdd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:24:20.386 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:20.384+0000 7f297bdd87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:24:20.387 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:20.384+0000 7f297bdd87c0 -1 bdev(0x559c358ea800 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-08-18T21:24:20.387 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:20.384+0000 7f297bdd87c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-08-18T21:24:26.634 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-08-18T21:24:26.634 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:24:26.635 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd1 key to auth repository 2024-08-18T21:24:26.636 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-18T21:24:26.636 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:24:27.220 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:24:27.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-18T21:24:27.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:24:27.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:24:27.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:24:27.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:24:27.260 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:27.256+0000 7fb75e4437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:24:27.288 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:27.284+0000 7fb75e4437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:24:27.290 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:27.288+0000 7fb75e4437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:24:27.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-18T21:24:27.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:24:27.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:24:27.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:24:27.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:24:27.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:27.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:24:27.688 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:24:27.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:27.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:28.106 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:28.331 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:28.328+0000 7fb75e4437c0 -1 Falling back to public interface 2024-08-18T21:24:29.109 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:29.109 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:29.109 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:24:29.110 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:24:29.110 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:29.110 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:29.595 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:30.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:30.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:30.598 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:24:30.598 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:24:30.598 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:30.598 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:31.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:31.212 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:31.208+0000 7fb75e4437c0 -1 osd.1 0 log_to_monitors true 2024-08-18T21:24:32.054 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:32.054 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:32.054 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:24:32.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:24:32.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:32.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:32.610 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:33.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:33.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:33.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:24:33.612 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:24:33.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:33.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:34.078 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:35.080 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:35.080 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:35.080 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:24:35.081 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:24:35.081 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:35.081 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:35.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:36.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:36.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:36.526 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:24:36.527 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:24:36.527 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:36.527 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:36.995 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:37.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:37.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:37.997 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:24:37.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:24:37.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:37.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:38.752 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:39.754 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:39.754 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:39.754 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:24:39.755 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:24:39.755 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:39.755 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:40.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:41.359 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:41.359 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:41.360 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:24:41.360 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:24:41.360 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:41.360 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:41.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:42.827 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:42.827 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:24:42.828 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:42.828 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:24:42.828 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:42.828 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:43.359 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:44.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:44.362 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:24:44.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:44.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:24:44.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:44.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:45.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:46.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:46.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:46.375 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:24:46.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:24:46.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:46.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:47.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:48.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:48.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:48.139 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:24:48.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:24:48.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:48.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:49.686 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:50.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:50.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:50.688 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:24:50.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:24:50.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:50.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:51.610 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:52.611 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:52.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:52.612 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:24:52.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:24:52.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:52.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:53.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:54.729 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:54.729 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:54.729 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:24:54.729 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:24:54.730 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:54.730 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:55.530 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:24:56.532 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:24:56.532 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:24:56.532 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:24:56.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:24:56.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:24:56.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:24:57.046 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1189025801,v1:127.0.0.1:6811/1189025801] [v2:127.0.0.1:6812/1189025801,v1:127.0.0.1:6813/1189025801] exists,up 57cc8ad2-e869-467a-9640-a317277ee238 2024-08-18T21:24:57.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:24:57.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:24:57.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:24:57.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-08-18T21:24:57.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-08-18T21:24:57.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:24:57.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:24:57.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-18T21:24:57.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:24:57.050 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:24:57.051 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:24:57.051 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:24:57.051 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:24:57.051 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:24:57.051 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:24:57.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:24:57.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-08-18T21:24:57.053 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:24:57.054 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e59c966a-f886-4300-b1de-6e9155227bbc 2024-08-18T21:24:57.054 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e59c966a-f886-4300-b1de-6e9155227bbc' 2024-08-18T21:24:57.054 INFO:tasks.workunit.client.0.smithi099.stdout:add osd2 e59c966a-f886-4300-b1de-6e9155227bbc 2024-08-18T21:24:57.054 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:24:57.083 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpZsJmz7ruBBAAUeqJ/JVt0gdco2zvbE1amQ== 2024-08-18T21:24:57.083 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpZsJmz7ruBBAAUeqJ/JVt0gdco2zvbE1amQ=="}' 2024-08-18T21:24:57.084 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e59c966a-f886-4300-b1de-6e9155227bbc -i td/divergent-priors/2/new.json 2024-08-18T21:24:57.627 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:24:57.643 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-08-18T21:24:57.644 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpZsJmz7ruBBAAUeqJ/JVt0gdco2zvbE1amQ== --osd-uuid e59c966a-f886-4300-b1de-6e9155227bbc 2024-08-18T21:24:57.694 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:57.691+0000 7f340b5527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:24:57.708 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:57.703+0000 7f340b5527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:24:57.710 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:57.707+0000 7f340b5527c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:24:57.710 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:57.707+0000 7f340b5527c0 -1 bdev(0x55eaa5cb3c00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-08-18T21:24:57.710 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:24:57.707+0000 7f340b5527c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-08-18T21:25:04.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-08-18T21:25:04.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:25:04.195 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd2 key to auth repository 2024-08-18T21:25:04.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-18T21:25:04.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:25:04.793 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:25:04.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-18T21:25:04.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:25:04.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:25:04.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:25:04.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:25:04.826 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:25:04.823+0000 7fb7d8e517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:25:04.847 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:25:04.843+0000 7fb7d8e517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:25:04.849 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:25:04.847+0000 7fb7d8e517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:25:05.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-18T21:25:05.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:25:05.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:25:05.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:25:05.258 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:25:05.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:25:05.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:05.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:25:05.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:05.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:05.681 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:06.087 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:25:06.083+0000 7fb7d8e517c0 -1 Falling back to public interface 2024-08-18T21:25:06.683 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:06.683 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:06.683 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:25:06.683 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:25:06.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:06.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:07.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:08.182 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:08.182 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:08.182 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:25:08.183 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:25:08.183 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:08.183 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:08.670 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:08.989 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:25:08.987+0000 7fb7d8e517c0 -1 osd.2 0 log_to_monitors true 2024-08-18T21:25:09.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:09.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:09.672 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:25:09.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:25:09.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:09.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:10.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:11.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:11.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:11.363 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:25:11.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:25:11.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:11.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:11.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:12.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:12.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:12.904 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:25:12.904 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:25:12.904 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:12.904 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:13.337 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:14.339 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:14.339 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:14.339 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:25:14.340 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:25:14.340 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:14.340 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:14.905 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:15.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:15.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:15.906 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:25:15.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:25:15.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:15.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:16.429 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:17.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:17.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:17.431 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:25:17.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:25:17.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:17.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:17.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:18.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:18.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:18.979 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:25:18.980 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:25:18.980 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:18.980 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:19.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:20.530 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:20.530 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:20.530 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:25:20.531 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:25:20.531 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:20.531 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:21.063 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:22.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:22.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:22.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:25:22.066 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:25:22.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:22.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:22.575 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:23.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:23.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:23.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:25:23.577 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:25:23.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:23.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:24.394 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:25.395 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:25.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:25.396 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:25:25.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:25:25.397 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:25.397 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:26.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:27.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:27.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:27.195 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:25:27.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:25:27.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:27.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:27.821 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:28.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:28.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:28.823 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:25:28.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:25:28.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:28.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:29.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:30.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:30.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:30.926 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:25:30.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:25:30.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:30.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:31.739 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:32.741 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:32.741 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:32.742 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:25:32.742 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:25:32.742 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:32.742 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:33.692 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:25:34.694 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:25:34.694 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:25:34.694 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:25:34.695 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:25:34.695 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:25:34.695 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:25:35.203 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3634747281,v1:127.0.0.1:6819/3634747281] [v2:127.0.0.1:6820/3634747281,v1:127.0.0.1:6821/3634747281] exists,up e59c966a-f886-4300-b1de-6e9155227bbc 2024-08-18T21:25:35.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:25:35.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:25:35.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:25:35.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-08-18T21:25:36.073 INFO:tasks.workunit.client.0.smithi099.stderr:noout is set 2024-08-18T21:25:36.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-08-18T21:25:36.801 INFO:tasks.workunit.client.0.smithi099.stderr:noin is set 2024-08-18T21:25:36.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-08-18T21:25:37.572 INFO:tasks.workunit.client.0.smithi099.stderr:nodown is set 2024-08-18T21:25:37.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-08-18T21:25:37.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-18T21:25:38.205 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'test' already exists 2024-08-18T21:25:38.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-18T21:25:39.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-08-18T21:25:40.109 INFO:tasks.workunit.client.0.smithi099.stderr:set pool 1 size to 3 2024-08-18T21:25:40.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-08-18T21:25:40.849 INFO:tasks.workunit.client.0.smithi099.stderr:set pool 1 min_size to 2 2024-08-18T21:25:40.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-08-18T21:25:40.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:25:40.874 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:25:41.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:25:41.411 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:25:41.411 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:25:41.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:25:41.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:25:41.412 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:25:41.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836506 2024-08-18T21:25:41.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836506 2024-08-18T21:25:41.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836506' 2024-08-18T21:25:41.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:25:41.590 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:25:41.751 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672973 2024-08-18T21:25:41.751 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672973 2024-08-18T21:25:41.751 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836506 1-42949672973' 2024-08-18T21:25:41.752 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:25:41.752 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:25:41.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509450 2024-08-18T21:25:41.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509450 2024-08-18T21:25:41.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836506 1-42949672973 2-64424509450' 2024-08-18T21:25:41.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:25:41.915 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-08-18T21:25:41.916 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:25:41.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:25:41.917 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836506 2024-08-18T21:25:41.917 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:25:41.918 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836506 2024-08-18T21:25:41.919 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836506 2024-08-18T21:25:41.919 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-08-18T21:25:41.919 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:25:42.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836506 -lt 21474836506 2024-08-18T21:25:42.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:25:42.375 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672973 2024-08-18T21:25:42.376 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:25:42.377 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:25:42.378 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672973 2024-08-18T21:25:42.378 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:25:42.379 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672973 2024-08-18T21:25:42.379 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672973 2024-08-18T21:25:42.380 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-08-18T21:25:42.380 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:25:42.871 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672973 -lt 42949672973 2024-08-18T21:25:42.871 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:25:42.872 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-08-18T21:25:42.872 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:25:42.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:25:42.874 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509450 2024-08-18T21:25:42.874 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:25:42.875 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509450 2024-08-18T21:25:42.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-08-18T21:25:42.876 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509450 2024-08-18T21:25:42.876 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:25:43.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509448 -lt 64424509450 2024-08-18T21:25:43.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:25:44.377 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:25:44.378 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:25:44.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509450 -lt 64424509450 2024-08-18T21:25:44.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-08-18T21:25:44.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:25:44.815 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:25:44.815 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:25:44.815 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-18T21:25:44.816 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:25:44.816 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:25:44.816 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:25:44.816 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:25:44.816 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:25:44.919 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-18T21:25:44.919 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:25:44.919 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:25:44.919 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:25:44.919 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:25:44.919 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:25:45.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:25:45.374 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:25:45.374 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:25:45.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:25:45.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:25:45.375 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:25:45.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836509 2024-08-18T21:25:45.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836509 2024-08-18T21:25:45.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836509' 2024-08-18T21:25:45.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:25:45.560 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:25:45.717 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672976 2024-08-18T21:25:45.718 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672976 2024-08-18T21:25:45.718 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836509 1-42949672976' 2024-08-18T21:25:45.718 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:25:45.718 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:25:45.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509453 2024-08-18T21:25:45.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509453 2024-08-18T21:25:45.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836509 1-42949672976 2-64424509453' 2024-08-18T21:25:45.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:25:45.869 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836509 2024-08-18T21:25:45.869 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:25:45.870 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:25:45.871 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836509 2024-08-18T21:25:45.871 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:25:45.872 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836509 2024-08-18T21:25:45.872 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-08-18T21:25:45.872 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836509 2024-08-18T21:25:45.872 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:25:46.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836509 -lt 21474836509 2024-08-18T21:25:46.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:25:46.336 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672976 2024-08-18T21:25:46.337 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:25:46.339 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:25:46.340 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672976 2024-08-18T21:25:46.340 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:25:46.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672976 2024-08-18T21:25:46.341 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672976 2024-08-18T21:25:46.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-08-18T21:25:46.342 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:25:46.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672976 -lt 42949672976 2024-08-18T21:25:46.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:25:46.797 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509453 2024-08-18T21:25:46.797 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:25:46.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:25:46.799 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:25:46.800 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509453 2024-08-18T21:25:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509453 2024-08-18T21:25:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-08-18T21:25:46.800 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509453 2024-08-18T21:25:46.801 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:25:47.338 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509451 -lt 64424509453 2024-08-18T21:25:47.338 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:25:48.340 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:25:48.341 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:25:48.770 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509453 -lt 64424509453 2024-08-18T21:25:48.771 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:25:48.771 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:25:48.772 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:25:49.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:25:49.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:25:49.375 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:25:49.375 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:25:49.375 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:25:49.375 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:25:49.375 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:25:49.375 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:25:49.791 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-18T21:25:49.791 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:25:49.793 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:25:49.793 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:25:50.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-18T21:25:50.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:25:50.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:25:50.345 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-08-18T21:25:50.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-08-18T21:25:50.745 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:25:50.761 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-08-18T21:25:50.761 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-08-18T21:25:50.761 INFO:tasks.workunit.client.0.smithi099.stdout:primary and soon to be divergent is 1 2024-08-18T21:25:50.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-08-18T21:25:51.156 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:25:51.157 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:25:39.265236+0000 0'0 29:22 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-19T21:55:39.796353+0000 0 0 2024-08-18T21:25:51.157 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:25:51.157 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:25:51.157 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:25:51.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-08-18T21:25:51.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-08-18T21:25:51.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-08-18T21:25:51.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-08-18T21:25:51.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-08-18T21:25:51.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-08-18T21:25:51.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-08-18T21:25:51.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-08-18T21:25:51.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-08-18T21:25:51.173 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-08-18T21:25:51.173 INFO:tasks.workunit.client.0.smithi099.stdout:writing initial objects 2024-08-18T21:25:51.173 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-08-18T21:25:51.173 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-08-18T21:25:51.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:51.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-08-18T21:25:51.338 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:51.338 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-08-18T21:25:51.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:51.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-08-18T21:25:51.674 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:51.674 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-08-18T21:25:51.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:51.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-08-18T21:25:52.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:52.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-08-18T21:25:52.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:52.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-08-18T21:25:52.386 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:52.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-08-18T21:25:52.554 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:52.554 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-08-18T21:25:52.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:52.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-08-18T21:25:52.955 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:52.955 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-08-18T21:25:53.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:53.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-08-18T21:25:53.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:53.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-08-18T21:25:53.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:53.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-08-18T21:25:53.650 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:53.650 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-08-18T21:25:53.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:53.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-08-18T21:25:53.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:53.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-08-18T21:25:54.145 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:54.145 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-08-18T21:25:54.328 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:54.329 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-08-18T21:25:54.505 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:54.505 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-08-18T21:25:54.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:54.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-08-18T21:25:54.865 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:54.865 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-08-18T21:25:55.041 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:55.041 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-08-18T21:25:55.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:55.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-08-18T21:25:55.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:55.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-08-18T21:25:55.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:55.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-08-18T21:25:55.686 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:55.686 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-08-18T21:25:55.853 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:55.854 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-08-18T21:25:56.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:56.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-08-18T21:25:56.280 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:56.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-08-18T21:25:56.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:56.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-08-18T21:25:56.683 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:56.683 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-08-18T21:25:56.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:56.842 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-08-18T21:25:56.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:56.993 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-08-18T21:25:57.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:57.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-08-18T21:25:57.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:57.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-08-18T21:25:57.478 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:57.478 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-08-18T21:25:57.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:57.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-08-18T21:25:57.846 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:57.846 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-08-18T21:25:58.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:58.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-08-18T21:25:58.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:58.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-08-18T21:25:58.400 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:58.400 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-08-18T21:25:58.559 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:58.559 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-08-18T21:25:58.719 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:58.720 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-08-18T21:25:58.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:58.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-08-18T21:25:59.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:59.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-08-18T21:25:59.220 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:59.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-08-18T21:25:59.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:59.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-08-18T21:25:59.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:59.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-08-18T21:25:59.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:59.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-08-18T21:25:59.957 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:25:59.957 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-08-18T21:26:00.125 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:00.125 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-08-18T21:26:00.300 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:00.300 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-08-18T21:26:00.486 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:00.486 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-08-18T21:26:00.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:00.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-08-18T21:26:00.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:00.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-08-18T21:26:01.030 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:01.030 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-08-18T21:26:01.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:01.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-08-18T21:26:01.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:01.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-08-18T21:26:01.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:01.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-08-18T21:26:01.708 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:01.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-08-18T21:26:01.866 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:01.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-08-18T21:26:02.061 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:02.061 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-08-18T21:26:02.227 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:02.227 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-08-18T21:26:02.428 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:02.429 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-08-18T21:26:02.605 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:02.605 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-08-18T21:26:02.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:02.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-08-18T21:26:02.981 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:02.982 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-08-18T21:26:03.149 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:03.149 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-08-18T21:26:03.316 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:03.316 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-08-18T21:26:03.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:03.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-08-18T21:26:03.635 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:03.635 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-08-18T21:26:03.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:03.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-08-18T21:26:03.994 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:03.994 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-08-18T21:26:04.162 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:04.163 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-08-18T21:26:04.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:04.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-08-18T21:26:04.523 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:04.523 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-08-18T21:26:04.691 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:04.691 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-08-18T21:26:04.849 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:04.849 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-08-18T21:26:05.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:05.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-08-18T21:26:05.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:05.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-08-18T21:26:05.335 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:05.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-08-18T21:26:05.511 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:05.511 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-08-18T21:26:05.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:05.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-08-18T21:26:05.863 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:05.863 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-08-18T21:26:06.089 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:06.090 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-08-18T21:26:06.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:06.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-08-18T21:26:06.466 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:06.466 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-08-18T21:26:06.634 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:06.634 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-08-18T21:26:06.793 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:06.793 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-08-18T21:26:06.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:06.954 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-08-18T21:26:07.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:07.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-08-18T21:26:07.295 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:07.295 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-08-18T21:26:07.480 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:07.480 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-08-18T21:26:07.664 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:07.667 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-08-18T21:26:07.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:07.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-08-18T21:26:08.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:08.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-08-18T21:26:08.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:08.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-08-18T21:26:08.392 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:08.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-08-18T21:26:08.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-08-18T21:26:08.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-08-18T21:26:08.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-18T21:26:08.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-08-18T21:26:08.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:26:08.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:26:08.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:26:08.754 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-18T21:26:08.755 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:26:08.755 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:26:08.755 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:26:08.755 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:26:08.755 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:26:08.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-18T21:26:08.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:26:08.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:26:08.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:26:08.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:26:08.832 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:26:09.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:26:09.271 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:26:09.271 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:26:09.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:26:09.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:26:09.272 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:26:09.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836515 2024-08-18T21:26:09.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836515 2024-08-18T21:26:09.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836515' 2024-08-18T21:26:09.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:26:09.433 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:26:09.595 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672983 2024-08-18T21:26:09.595 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672983 2024-08-18T21:26:09.595 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836515 1-42949672983' 2024-08-18T21:26:09.595 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:26:09.595 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:26:09.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509459 2024-08-18T21:26:09.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509459 2024-08-18T21:26:09.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836515 1-42949672983 2-64424509459' 2024-08-18T21:26:09.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:26:09.760 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836515 2024-08-18T21:26:09.760 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:26:09.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:26:09.763 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836515 2024-08-18T21:26:09.763 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:26:09.764 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836515 2024-08-18T21:26:09.765 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836515 2024-08-18T21:26:09.765 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-08-18T21:26:09.765 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:26:10.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836515 -lt 21474836515 2024-08-18T21:26:10.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:26:10.188 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-08-18T21:26:10.188 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:26:10.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:26:10.191 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672983 2024-08-18T21:26:10.191 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:26:10.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672983 2024-08-18T21:26:10.192 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672983 2024-08-18T21:26:10.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-08-18T21:26:10.193 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:26:10.700 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672983 -lt 42949672983 2024-08-18T21:26:10.700 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:26:10.701 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509459 2024-08-18T21:26:10.701 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:26:10.702 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:26:10.703 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509459 2024-08-18T21:26:10.704 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:26:10.705 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509459 2024-08-18T21:26:10.705 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509459 2024-08-18T21:26:10.705 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-08-18T21:26:10.705 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:26:11.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509459 -lt 64424509459 2024-08-18T21:26:11.131 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:26:11.131 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:26:11.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:26:11.699 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:26:11.699 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:26:11.700 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:26:11.700 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:26:11.700 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:26:11.700 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:26:11.700 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:26:11.700 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:26:12.107 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-18T21:26:12.107 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:26:12.108 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:26:12.108 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:26:12.726 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-18T21:26:12.726 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:26:12.726 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:26:12.727 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-08-18T21:26:12.727 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-08-18T21:26:12.727 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-08-18T21:26:12.728 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-18T21:26:12.728 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-08-18T21:26:13.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-08-18T21:26:13.245 INFO:tasks.workunit.client.0.smithi099.stdout:blackholing osds 0 2 2024-08-18T21:26:13.246 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-08-18T21:26:13.246 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-08-18T21:26:13.637 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:26:13.637 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-08-18T21:25:39.265236+0000 29'100 29:122 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-19T21:55:39.796353+0000 0 0 2024-08-18T21:26:13.637 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:26:13.637 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:26:13.637 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:26:13.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-08-18T21:26:13.653 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-08-18T21:26:13.653 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-18T21:26:13.653 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-18T21:26:13.654 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:26:13.654 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:26:13.655 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:26:13.655 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.0.asok 2024-08-18T21:26:13.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-08-18T21:26:13.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.0.asok config set objectstore_blackhole 1 2024-08-18T21:26:13.764 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:26:13.764 INFO:tasks.workunit.client.0.smithi099.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-18T21:26:13.764 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:26:13.775 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-08-18T21:26:13.777 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-08-18T21:26:13.777 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-18T21:26:13.777 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-18T21:26:13.777 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:26:13.777 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:26:13.777 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:26:13.778 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.2.asok 2024-08-18T21:26:13.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-08-18T21:26:13.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.2.asok config set objectstore_blackhole 1 2024-08-18T21:26:13.883 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:26:13.883 INFO:tasks.workunit.client.0.smithi099.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-18T21:26:13.883 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:26:13.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-08-18T21:26:13.896 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-08-18T21:26:13.897 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-08-18T21:26:13.898 INFO:tasks.workunit.client.0.smithi099.stdout:writing divergent object 2024-08-18T21:26:13.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-08-18T21:26:13.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-08-18T21:26:13.898 INFO:tasks.workunit.client.0.smithi099.stdout:create missing divergent object 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: local pooltype=rep 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: shift 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: local which=data 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: shift 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: local poolname=test 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: shift 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: local objname=existing_99 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: shift 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: local dir=td/divergent-priors 2024-08-18T21:26:13.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: shift 2024-08-18T21:26:13.904 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: local shard_id=0 2024-08-18T21:26:13.904 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: inject_eio: shift 2024-08-18T21:26:13.904 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: get_osds test existing_99 2024-08-18T21:26:13.904 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-08-18T21:26:13.904 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2024-08-18T21:26:13.904 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-08-18T21:26:13.904 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2024-08-18T21:26:14.354 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-08-18T21:26:14.355 INFO:tasks.workunit.client.0.smithi099.stderr:0 2024-08-18T21:26:14.355 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:26:14.355 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2024-08-18T21:26:14.355 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: initial_osds=('1' '0' '2') 2024-08-18T21:26:14.355 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local -a initial_osds 2024-08-18T21:26:14.355 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: local osd_id=1 2024-08-18T21:26:14.355 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: '[' rep '!=' ec ']' 2024-08-18T21:26:14.355 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: inject_eio: shard_id= 2024-08-18T21:26:14.356 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: cat td/divergent-priors/1/type 2024-08-18T21:26:14.356 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: type=bluestore 2024-08-18T21:26:14.356 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-08-18T21:26:14.356 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-08-18T21:26:14.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-08-18T21:26:14.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-08-18T21:26:14.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-08-18T21:26:14.357 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-08-18T21:26:14.357 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-08-18T21:26:14.357 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-18T21:26:14.358 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-18T21:26:14.358 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:26:14.358 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:26:14.359 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:26:14.359 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.1.asok 2024-08-18T21:26:14.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.27423/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-08-18T21:26:14.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-08-18T21:26:14.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: local loop=0 2024-08-18T21:26:14.480 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2433: inject_eio: grep -q Invalid 2024-08-18T21:26:14.480 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: get_asok_path osd.1 2024-08-18T21:26:14.481 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-18T21:26:14.481 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-18T21:26:14.481 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:26:14.481 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:26:14.481 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:26:14.481 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.1.asok 2024-08-18T21:26:14.482 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: CEPH_ARGS= 2024-08-18T21:26:14.482 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.1.asok injectdataerr test existing_99 2024-08-18T21:26:14.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-08-18T21:26:14.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-08-18T21:26:24.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-08-18T21:26:24.612 INFO:tasks.workunit.client.0.smithi099.stderr:rados: no process found 2024-08-18T21:26:24.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-08-18T21:26:24.613 INFO:tasks.workunit.client.0.smithi099.stdout:killing all the osds 2024-08-18T21:26:24.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-08-18T21:26:25.019 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:26:25.019 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 active+clean 2024-08-18T21:26:14.754024+0000 29'101 29:131 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-19T21:55:39.796353+0000 0 0 2024-08-18T21:26:25.019 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:26:25.019 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:26:25.020 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:26:25.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-08-18T21:26:25.036 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:26:25.036 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:26:25.036 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:26:25.036 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:26:25.036 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:26:25.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:26:25.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-08-18T21:26:25.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-08-18T21:26:25.984 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already down. 2024-08-18T21:26:26.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-08-18T21:26:26.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-08-18T21:26:26.652 INFO:tasks.workunit.client.0.smithi099.stderr:osd.1 is already down. 2024-08-18T21:26:26.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-08-18T21:26:26.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-08-18T21:26:27.224 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already down. 2024-08-18T21:26:27.240 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-08-18T21:26:27.240 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-08-18T21:26:27.834 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already out. 2024-08-18T21:26:27.850 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-08-18T21:26:27.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-08-18T21:26:28.541 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already out. 2024-08-18T21:26:28.557 INFO:tasks.workunit.client.0.smithi099.stdout:bringing up non_divergent 0 2 2024-08-18T21:26:28.558 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-08-18T21:26:28.558 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-08-18T21:26:28.981 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:26:28.981 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 stale+active+clean 2024-08-18T21:26:14.754024+0000 29'101 29:131 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-19T21:55:39.796353+0000 0 0 2024-08-18T21:26:28.981 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:26:28.981 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:26:28.982 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:26:29.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-08-18T21:26:29.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-08-18T21:26:29.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:26:29.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:26:29.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-18T21:26:29.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:26:29.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-18T21:26:29.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:26:29.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:26:29.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:26:29.001 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:26:29.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:26:29.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:26:29.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:26:29.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:26:29.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:26:29.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:26:29.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:26:29.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:26:29.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:26:29.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:26:29.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-18T21:26:29.003 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:26:29.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-18T21:26:29.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:26:29.004 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-18T21:26:29.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-18T21:26:29.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:26:29.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:26:29.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:26:29.040 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:26:29.038+0000 7f3713a4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:26:29.042 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:26:29.038+0000 7f3713a4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:26:29.044 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:26:29.038+0000 7f3713a4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:26:29.465 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-18T21:26:29.466 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:26:29.466 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:26:29.466 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:26:29.466 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:26:29.466 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:26:29.467 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:26:29.467 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:26:29.467 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:26:29.467 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:26:29.949 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:26:30.050 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:26:30.046+0000 7f3713a4c7c0 -1 Falling back to public interface 2024-08-18T21:26:30.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:26:30.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:26:30.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:26:30.951 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:26:30.952 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:26:30.952 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:26:31.401 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:26:32.403 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:26:32.403 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:26:32.403 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:26:32.404 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:26:32.404 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:26:32.404 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:26:32.858 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:26:33.441 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:26:33.438+0000 7f3713a4c7c0 -1 osd.0 29 log_to_monitors true 2024-08-18T21:26:33.860 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:26:33.860 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:26:33.860 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:26:33.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:26:33.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:26:33.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:26:34.403 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/2203422713,v1:127.0.0.1:6803/2203422713] [v2:127.0.0.1:6804/2203422713,v1:127.0.0.1:6805/2203422713] exists,up 81ca0476-b29b-4284-91e8-459871ca9868 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:26:34.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:26:34.408 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:26:34.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-08-18T21:26:34.410 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:26:34.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-18T21:26:34.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:26:34.411 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-08-18T21:26:34.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-18T21:26:34.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:26:34.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:26:34.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:26:34.443 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:26:34.438+0000 7f43c04057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:26:34.475 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:26:34.470+0000 7f43c04057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:26:34.477 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:26:34.474+0000 7f43c04057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:26:34.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-18T21:26:34.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:26:34.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:26:34.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:26:34.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:26:34.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:26:34.940 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:26:34.941 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:26:34.941 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:26:34.941 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:26:35.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:26:35.506 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:26:35.502+0000 7f43c04057c0 -1 Falling back to public interface 2024-08-18T21:26:36.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:26:36.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:26:36.432 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:26:36.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:26:36.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:26:36.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:26:36.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:26:37.860 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:26:37.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:26:37.861 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:26:37.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:26:37.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:26:37.862 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:26:38.316 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:26:38.736 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:26:38.729+0000 7f43c04057c0 -1 osd.2 29 log_to_monitors true 2024-08-18T21:26:39.317 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:26:39.317 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:26:39.318 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:26:39.318 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:26:39.318 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:26:39.318 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:26:39.860 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2422497712,v1:127.0.0.1:6811/2422497712] [v2:127.0.0.1:6812/2422497712,v1:127.0.0.1:6813/2422497712] exists,up e59c966a-f886-4300-b1de-6e9155227bbc 2024-08-18T21:26:39.860 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:26:39.860 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:26:39.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:26:39.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-08-18T21:26:39.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-08-18T21:26:40.524 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already in. 2024-08-18T21:26:40.542 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-08-18T21:26:40.542 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-08-18T21:26:41.089 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already in. 2024-08-18T21:26:41.105 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-18T21:26:41.105 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-08-18T21:26:41.105 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:26:41.105 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:26:41.105 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:26:41.106 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-18T21:26:41.106 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:26:41.106 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:26:41.106 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:26:41.107 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:26:41.107 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:26:41.173 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-18T21:26:41.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:26:41.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:26:41.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:26:41.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:26:41.174 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:26:41.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:26:41.623 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:26:41.623 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:26:41.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:26:41.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:26:41.624 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:26:41.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=154618822660 2024-08-18T21:26:41.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 154618822660 2024-08-18T21:26:41.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-154618822660' 2024-08-18T21:26:41.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:26:41.801 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:26:41.949 INFO:tasks.workunit.client.0.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-18T21:26:41.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq= 2024-08-18T21:26:41.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z '' 2024-08-18T21:26:41.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: continue 2024-08-18T21:26:41.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:26:41.951 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:26:42.124 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=163208757251 2024-08-18T21:26:42.124 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 163208757251 2024-08-18T21:26:42.125 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-08-18T21:26:42.125 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:26:42.125 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822660 2024-08-18T21:26:42.125 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:26:42.126 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:26:42.126 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-154618822660 2024-08-18T21:26:42.127 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:26:42.129 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 154618822660 2024-08-18T21:26:42.129 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=154618822660 2024-08-18T21:26:42.129 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-08-18T21:26:42.129 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:26:42.644 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 154618822660 -lt 154618822660 2024-08-18T21:26:42.644 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:26:42.645 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757251 2024-08-18T21:26:42.645 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:26:42.647 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:26:42.648 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-163208757251 2024-08-18T21:26:42.648 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:26:42.649 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=163208757251 2024-08-18T21:26:42.649 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-08-18T21:26:42.649 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 163208757251 2024-08-18T21:26:42.650 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:26:43.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 1 -lt 163208757251 2024-08-18T21:26:43.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:26:44.068 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:26:44.069 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:26:44.518 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 163208757251 -lt 163208757251 2024-08-18T21:26:44.519 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:26:44.519 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:26:44.519 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:26:45.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:26:45.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:26:45.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:26:45.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:26:45.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:26:45.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:26:45.133 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:26:45.133 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:26:45.571 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:26:45.572 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:26:45.572 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:26:45.572 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:26:46.138 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:26:46.138 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-18T21:26:46.138 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-18T21:26:46.138 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-18T21:26:46.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:26:46.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-18T21:26:46.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:26:46.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:26:46.240 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:26:46.240 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:26:46.240 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:26:46.240 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:26:46.241 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:26:46.241 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:26:46.696 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:26:46.696 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:26:46.697 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:26:46.697 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:26:47.244 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:26:47.244 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:26:47.244 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:26:47.244 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:26:47.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:26:47.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:26:47.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:26:47.245 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:26:47.245 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:26:47.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:26:47.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:26:47.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-18T21:26:47.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:26:47.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-18T21:26:47.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:26:47.999 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:26:47.999 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:26:47.999 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:26:47.999 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:26:48.000 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:26:48.000 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:26:48.000 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:26:48.417 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:26:48.418 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:26:48.419 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:26:48.419 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:26:48.987 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:26:48.988 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:26:48.988 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:26:48.988 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:26:48.988 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:26:48.988 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:26:48.988 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:26:48.988 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:26:48.988 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:26:49.553 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:26:49.553 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:26:49.553 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-18T21:26:49.553 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:26:49.553 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-18T21:26:49.954 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:26:49.955 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:26:49.955 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:26:49.955 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:26:49.955 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:26:49.955 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:26:49.956 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:26:49.956 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:26:50.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:26:50.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:26:50.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:26:50.372 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:26:50.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:26:50.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:26:50.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:26:50.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:26:50.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:26:50.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:26:50.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:26:50.907 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:26:50.907 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:26:51.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:26:51.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:26:51.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-18T21:26:51.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:26:51.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-18T21:26:52.296 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:26:52.296 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:26:52.297 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:26:52.297 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:26:52.297 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:26:52.297 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:26:52.298 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:26:52.298 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:26:52.733 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:26:52.734 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:26:52.734 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:26:52.734 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:26:53.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:26:53.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:26:53.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:26:53.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:26:53.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:26:53.303 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:26:53.303 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:26:53.303 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:26:53.303 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:26:53.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:26:53.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:26:53.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-18T21:26:53.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:26:53.899 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-18T21:26:55.499 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:26:55.499 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:26:55.500 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:26:55.500 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:26:55.500 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:26:55.500 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:26:55.500 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:26:55.501 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:26:55.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:26:55.918 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:26:55.919 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:26:55.919 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:26:56.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:26:56.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:26:56.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:26:56.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:26:56.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:26:56.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:26:56.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:26:56.525 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:26:56.525 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:26:57.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:26:57.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:26:57.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-18T21:26:57.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:26:57.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-18T21:27:00.318 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:27:00.321 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:27:00.322 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:27:00.322 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:27:00.322 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:27:00.322 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:27:00.323 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:27:00.323 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:27:00.758 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:27:00.759 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:27:00.759 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:27:00.759 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:27:01.312 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:27:01.312 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:27:01.312 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:27:01.312 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:27:01.313 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:27:01.313 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:27:01.313 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:27:01.313 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:27:01.313 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:27:01.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:27:01.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:27:01.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-18T21:27:01.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:27:01.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-18T21:27:08.295 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:27:08.295 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:27:08.296 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:27:08.296 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:27:08.296 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:27:08.296 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:27:08.296 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:27:08.296 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:27:08.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:27:08.725 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:27:08.725 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:27:08.725 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:27:09.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:27:09.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:27:09.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:27:09.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:27:09.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:27:09.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:27:09.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:27:09.311 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:27:09.311 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:27:09.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:27:09.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:27:09.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-18T21:27:09.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:27:09.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-18T21:27:17.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:27:17.178 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:27:17.178 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:27:17.178 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:27:17.178 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:27:17.179 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:27:17.179 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:27:17.179 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:27:17.627 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:27:17.628 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:27:17.628 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:27:17.628 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:27:18.207 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:27:18.207 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:27:18.207 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:27:18.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:27:18.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:27:18.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:27:18.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:27:18.208 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:27:18.208 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:27:18.789 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:27:18.789 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:27:18.789 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-18T21:27:18.789 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-18T21:27:19.333 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:27:19.334 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_fingerprint": "3aa88c48-48d6-4ea5-9186-e49ea8249140", 2024-08-18T21:27:19.334 INFO:tasks.workunit.client.0.smithi099.stdout: "version": "19.0.0-5847-g79766939", 2024-08-18T21:27:19.334 INFO:tasks.workunit.client.0.smithi099.stdout: "commit": "797669399b32d2915e64eb6d4b1257ae3c50f80b", 2024-08-18T21:27:19.334 INFO:tasks.workunit.client.0.smithi099.stdout: "timestamp": "2024-08-18T21:27:19.332817+0000", 2024-08-18T21:27:19.334 INFO:tasks.workunit.client.0.smithi099.stdout: "tag": "", 2024-08-18T21:27:19.334 INFO:tasks.workunit.client.0.smithi099.stdout: "health": { 2024-08-18T21:27:19.334 INFO:tasks.workunit.client.0.smithi099.stdout: "status": "HEALTH_WARN", 2024-08-18T21:27:19.334 INFO:tasks.workunit.client.0.smithi099.stdout: "checks": { 2024-08-18T21:27:19.334 INFO:tasks.workunit.client.0.smithi099.stdout: "OSDMAP_FLAGS": { 2024-08-18T21:27:19.334 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:27:19.334 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 17 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [], 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: "OSD_DOWN": { 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 osds down", 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:27:19.335 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "osd.1 (root=default,host=smithi099) is down" 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: "PG_DEGRADED": { 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.336 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:27:19.337 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "application not enabled on pool 'test'" 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: "mutes": [] 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_first_committed": 1, 2024-08-18T21:27:19.338 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_last_committed": 1, 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap": { 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:23:41.262172Z", 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:23:41.262172Z", 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release": 19, 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release_name": "squid", 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "election_strategy": 1, 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "disallowed_leaders: ": "", 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": false, 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "tiebreaker_mon": "", 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_ranks: ": "", 2024-08-18T21:27:19.339 INFO:tasks.workunit.client.0.smithi099.stdout: "features": { 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "persistent": [ 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "kraken", 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "luminous", 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "mimic", 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap-prune", 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "nautilus", 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "octopus", 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "pacific", 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "elector-pinging", 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "quincy", 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "reef", 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "squid" 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.340 INFO:tasks.workunit.client.0.smithi099.stdout: "optional": [] 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: "mons": [ 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: "rank": 0, 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "a", 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115", 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 0 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.341 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115/0", 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: "priority": 0, 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 0, 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_location": "{}" 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: "quorum": [ 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: 0 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.342 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap": { 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 41, 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:23:42.301370+0000", 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:26:41.352592+0000", 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "last_up_change": "2024-08-18T21:26:39.001593+0000", 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "last_in_change": "2024-08-18T21:26:40.915161+0000", 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_num": 5800384, 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_set": [ 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "nodown", 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "noin", 2024-08-18T21:27:19.343 INFO:tasks.workunit.client.0.smithi099.stdout: "noout", 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "pglog_hardlimit", 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snapdirs", 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "recovery_deletes", 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "sortbitwise" 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_version": 9, 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "full_ratio": 0.99000000953674316, 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_snapshot": "", 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_max": 1, 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "max_osd": 3, 2024-08-18T21:27:19.344 INFO:tasks.workunit.client.0.smithi099.stdout: "require_min_compat_client": "luminous", 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "min_compat_client": "jewel", 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "require_osd_release": "squid", 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "allow_crimson": false, 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "pools": [ 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_name": "test", 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "create_time": "2024-08-18T21:25:37.967670+0000", 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": 1, 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_names": "hashpspool", 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "size": 3, 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "min_size": 2, 2024-08-18T21:27:19.345 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_rule": 0, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "is_stretch_pool": false, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "object_hash": 2, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_autoscale_mode": "off", 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num": 1, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num": 1, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num_target": 1, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_target": 1, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_pending": 1, 2024-08-18T21:27:19.346 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pg_merge_meta": { 2024-08-18T21:27:19.347 INFO:tasks.workunit.client.0.smithi099.stdout: "source_pgid": "0.0", 2024-08-18T21:27:19.347 INFO:tasks.workunit.client.0.smithi099.stdout: "ready_epoch": 0, 2024-08-18T21:27:19.347 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_started": 0, 2024-08-18T21:27:19.347 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": 0, 2024-08-18T21:27:19.347 INFO:tasks.workunit.client.0.smithi099.stdout: "source_version": "0'0", 2024-08-18T21:27:19.347 INFO:tasks.workunit.client.0.smithi099.stdout: "target_version": "0'0" 2024-08-18T21:27:19.347 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.347 INFO:tasks.workunit.client.0.smithi099.stdout: "last_change": "29", 2024-08-18T21:27:19.347 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend": "0", 2024-08-18T21:27:19.348 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "auid": 0, 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_mode": "selfmanaged", 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_seq": 0, 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_epoch": 0, 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_snaps": [], 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps": "[]", 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_bytes": 0, 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_objects": 0, 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "tiers": [], 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "tier_of": -1, 2024-08-18T21:27:19.349 INFO:tasks.workunit.client.0.smithi099.stdout: "read_tier": -1, 2024-08-18T21:27:19.350 INFO:tasks.workunit.client.0.smithi099.stdout: "write_tier": -1, 2024-08-18T21:27:19.350 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_mode": "none", 2024-08-18T21:27:19.350 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_bytes": 0, 2024-08-18T21:27:19.350 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_objects": 0, 2024-08-18T21:27:19.350 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-18T21:27:19.350 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-18T21:27:19.350 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-18T21:27:19.350 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_flush_age": 0, 2024-08-18T21:27:19.350 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_evict_age": 0, 2024-08-18T21:27:19.350 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profile": "", 2024-08-18T21:27:19.350 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_params": { 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "none" 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_period": 0, 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_count": 0, 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: "use_gmt_hitset": true, 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_search_last_n": 0, 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: "grade_table": [], 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: "stripe_width": 0, 2024-08-18T21:27:19.351 INFO:tasks.workunit.client.0.smithi099.stdout: "expected_num_objects": 0, 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "fast_read": false, 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "options": {}, 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "application_metadata": {}, 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "read_balance": { 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "score_type": "Fair distribution", 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "score_acting": 2.9900000095367432, 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "score_stable": 2.9900000095367432, 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_score": 0.67000001668930054, 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "raw_score_acting": 2, 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "raw_score_stable": 2, 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity_weighted": 1, 2024-08-18T21:27:19.352 INFO:tasks.workunit.client.0.smithi099.stdout: "average_primary_affinity": 1, 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: "average_primary_affinity_weighted": 1 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: "osds": [ 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "81ca0476-b29b-4284-91e8-459871ca9868", 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:27:19.353 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 5, 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 29, 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 36, 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 40, 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 30, 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6802", 2024-08-18T21:27:19.354 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2203422713 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6803", 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2203422713 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.355 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6804", 2024-08-18T21:27:19.356 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2203422713 2024-08-18T21:27:19.356 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.356 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.356 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.356 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6805", 2024-08-18T21:27:19.356 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2203422713 2024-08-18T21:27:19.356 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.356 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.356 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.356 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:27:19.356 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6808", 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2203422713 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6809", 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2203422713 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.357 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6806", 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2203422713 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6807", 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2203422713 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.358 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6803/2203422713", 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6805/2203422713", 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2203422713", 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2203422713", 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:27:19.359 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "57cc8ad2-e869-467a-9640-a317277ee238", 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 0, 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 0, 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 0, 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 10, 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 23, 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 31, 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.360 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1189025801 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1189025801 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.361 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1189025801 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1189025801 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.362 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1189025801 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1189025801 2024-08-18T21:27:19.363 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1189025801 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:27:19.364 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1189025801 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/1189025801", 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/1189025801", 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1189025801", 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1189025801", 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: "exists" 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.365 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "e59c966a-f886-4300-b1de-6e9155227bbc", 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 15, 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 29, 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 38, 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 0, 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 32, 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:27:19.366 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2422497712 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2422497712 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.367 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2422497712 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2422497712 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.368 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2422497712 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2422497712 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.369 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2422497712 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2422497712 2024-08-18T21:27:19.370 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/2422497712", 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/2422497712", 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2422497712", 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2422497712", 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.371 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_xinfo": [ 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:26:25.732827+0000", 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:24:19.068600+0000", 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.372 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:26:26.410779+0000", 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:24:56.059119+0000", 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:26:27.049540+0000", 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:27:19.373 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:25:34.158200+0000", 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap": [], 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_items": [], 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_primaries": [], 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_temp": [], 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_temp": [], 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: "blocklist": {}, 2024-08-18T21:27:19.374 INFO:tasks.workunit.client.0.smithi099.stdout: "range_blocklist": {}, 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profiles": { 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: "default": { 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: "k": "2", 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: "m": "2", 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: "plugin": "jerasure", 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: "technique": "reed_sol_van" 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps_queue": [], 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: "new_removed_snaps": [], 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: "new_purged_snaps": [], 2024-08-18T21:27:19.375 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_node_flags": {}, 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: "device_class_flags": {}, 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": { 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_enabled": false, 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_bucket_count": 0, 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: "degraded_stretch_mode": 0, 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: "recovering_stretch_mode": 0, 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_bucket": 0 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_metadata": [ 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:27:19.376 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6804/2203422713,v1:127.0.0.1:6805/2203422713]", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:27:19.377 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:23:49.544376Z", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:27:19.378 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6802/2203422713,v1:127.0.0.1:6803/2203422713]", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2203422713,v1:127.0.0.1:6809/2203422713]", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2203422713,v1:127.0.0.1:6807/2203422713]", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:27:19.379 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/0", 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.380 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/1189025801,v1:127.0.0.1:6813/1189025801]", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:27:19.381 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:27:19.382 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-18T21:27:19.382 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:27:19.382 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:27:19.382 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:27:19.382 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:27:19.382 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:27:19.382 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:27:19.382 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:27:19.382 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:27:19.382 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:27:19.382 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:24:25.943140Z", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/1189025801,v1:127.0.0.1:6811/1189025801]", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1189025801,v1:127.0.0.1:6817/1189025801]", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1189025801,v1:127.0.0.1:6815/1189025801]", 2024-08-18T21:27:19.383 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/1", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:27:19.384 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/2422497712,v1:127.0.0.1:6813/2422497712]", 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:27:19.385 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:27:19.386 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:27:19.387 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:27:19.387 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:27:19.387 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:25:03.519905Z", 2024-08-18T21:27:19.387 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:27:19.387 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:27:19.387 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:27:19.387 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:27:19.387 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:27:19.387 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:27:19.387 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:27:19.387 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/2422497712,v1:127.0.0.1:6811/2422497712]", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2422497712,v1:127.0.0.1:6817/2422497712]", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2422497712,v1:127.0.0.1:6815/2422497712]", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:27:19.388 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/2", 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_clean_epochs": { 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: "min_last_epoch_clean": 0, 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": { 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: "per_pool": [] 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_epochs": [ 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.389 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 36 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 38 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_first_committed": 1, 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_last_committed": 41, 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: "crushmap": { 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": [ 2024-08-18T21:27:19.390 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.0", 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.1", 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.2", 2024-08-18T21:27:19.391 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:27:19.392 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.392 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.392 INFO:tasks.workunit.client.0.smithi099.stdout: "types": [ 2024-08-18T21:27:19.392 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.392 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 0, 2024-08-18T21:27:19.392 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd" 2024-08-18T21:27:19.392 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.392 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.392 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:27:19.392 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "host" 2024-08-18T21:27:19.392 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 2, 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "chassis" 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 3, 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "rack" 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 4, 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "row" 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.393 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 5, 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pdu" 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 6, 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pod" 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 7, 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "room" 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.394 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 8, 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "datacenter" 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 9, 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "zone" 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 10, 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "region" 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:27:19.395 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "root" 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: "buckets": [ 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -1, 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default", 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:27:19.396 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:27:19.397 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.397 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:27:19.397 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:27:19.397 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:27:19.397 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.397 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.397 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.397 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.397 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -2, 2024-08-18T21:27:19.397 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default~hdd", 2024-08-18T21:27:19.397 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.398 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099", 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:27:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.400 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099~hdd", 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:27:19.401 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.402 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: "rules": [ 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_id": 0, 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_name": "replicated_rule", 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: "steps": [ 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "take", 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: "item": -1, 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: "item_name": "default" 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.403 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "choose_firstn", 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 0, 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "osd" 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "emit" 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: "tunables": { 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_tries": 0, 2024-08-18T21:27:19.404 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_fallback_tries": 0, 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_total_tries": 50, 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_descend_once": 1, 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_vary_r": 1, 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_stable": 1, 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_descents": 100, 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_collision_tries": 100, 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "straw_calc_version": 1, 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "allowed_bucket_algs": 54, 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "profile": "jewel", 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_tunables": 1, 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "legacy_tunables": 0, 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "minimum_required_version": "jewel", 2024-08-18T21:27:19.405 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables": 1, 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables2": 1, 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v2_rules": 0, 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables3": 1, 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v3_rules": 0, 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v4_buckets": 1, 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables5": 1, 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v5_rules": 0, 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: "has_msr_rules": 0 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_args": {} 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.406 INFO:tasks.workunit.client.0.smithi099.stdout: "fsmap": { 2024-08-18T21:27:19.407 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:27:19.407 INFO:tasks.workunit.client.0.smithi099.stdout: "btime": "2024-08-18T21:23:42:268919+0000", 2024-08-18T21:27:19.407 INFO:tasks.workunit.client.0.smithi099.stdout: "default_fscid": -1, 2024-08-18T21:27:19.407 INFO:tasks.workunit.client.0.smithi099.stderr:report 1123531566 2024-08-18T21:27:19.407 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": { 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": {}, 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "ro_compat": {}, 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "incompat": { 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_1": "base v0.20", 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_2": "client writeable ranges", 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_3": "default file layouts on dirs", 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_4": "dir inode in separate object", 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_5": "mds uses versioned encoding", 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_8": "no anchor table", 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_9": "file layout v2", 2024-08-18T21:27:19.408 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_10": "snaprealm v2", 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_11": "minor log segments", 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_12": "quiesce subvolumes" 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_flags": { 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: "enable_multiple": true, 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: "ever_enabled_multiple": true 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: "standbys": [], 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: "filesystems": [] 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.409 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_first_committed": 1, 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_last_committed": 1, 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "auth": { 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 5, 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "num_secrets": 3 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 1, 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_active": 1, 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_unknown": 0, 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osd": 3, 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_sum": { 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:27:19.410 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 61700, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 100, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 0, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 300, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 100, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 100, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:27:19.411 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 100, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 100, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:27:19.412 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:27:19.413 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:27:19.413 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:27:19.413 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:27:19.413 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:27:19.413 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:27:19.413 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:27:19.413 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:27:19.413 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:27:19.413 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:27:19.414 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:27:19.414 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:27:19.414 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:27:19.414 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:27:19.414 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:27:19.414 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.414 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:27:19.414 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:27:19.414 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:27:19.414 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:27:19.414 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 0, 2024-08-18T21:27:19.415 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 0, 2024-08-18T21:27:19.415 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:27:19.415 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:27:19.415 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:27:19.415 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:27:19.415 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:27:19.415 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.415 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 16, 2024-08-18T21:27:19.415 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 16, 2024-08-18T21:27:19.415 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 2, 2024-08-18T21:27:19.415 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 2, 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 0 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum": { 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 3, 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83660, 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2112, 2024-08-18T21:27:19.416 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314489140, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036879360, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2162688, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 460116, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10396, 2024-08-18T21:27:19.417 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416932 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 1, 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:27:19.418 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum_by_class": { 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: "hdd": { 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 3, 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:27:19.419 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83660, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2112, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314489140, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036879360, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2162688, 2024-08-18T21:27:19.420 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 460116, 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10396, 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416932 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 1, 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:27:19.421 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.422 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_stats": [ 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "poolid": 1, 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 1, 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 61700, 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 100, 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 0, 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 300, 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 100, 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:27:19.423 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:27:19.424 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 100, 2024-08-18T21:27:19.424 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:27:19.424 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:27:19.424 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:27:19.424 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 100, 2024-08-18T21:27:19.424 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 100, 2024-08-18T21:27:19.424 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:27:19.424 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:27:19.424 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:27:19.424 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:27:19.424 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:27:19.425 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:27:19.426 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 1228800, 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 185100, 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 16, 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 16, 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 2, 2024-08-18T21:27:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 2, 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 3 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_stats": [ 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 154618822667 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 42949672986 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.428 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 163208757258 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_state": [ 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: "state": "active+undersized+degraded", 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 1 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_osd": [ 2024-08-18T21:27:19.429 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 1, 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.430 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.431 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:27:19.431 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:27:19.431 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 0, 2024-08-18T21:27:19.431 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:27:19.431 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.431 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.431 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": [ 2024-08-18T21:27:19.431 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:27:19.431 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:27:19.431 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": {} 2024-08-18T21:27:19.431 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: "servicemap": { 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 5, 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:26:35.852617+0000", 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: "services": { 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": { 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: "daemons": { 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": "", 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: "0": { 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: "start_epoch": 0, 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: "start_stamp": "0.000000", 2024-08-18T21:27:19.432 INFO:tasks.workunit.client.0.smithi099.stdout: "gid": 0, 2024-08-18T21:27:19.433 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "(unrecognized address family 0)/0", 2024-08-18T21:27:19.433 INFO:tasks.workunit.client.0.smithi099.stdout: "metadata": {}, 2024-08-18T21:27:19.433 INFO:tasks.workunit.client.0.smithi099.stdout: "task_status": {} 2024-08-18T21:27:19.433 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.433 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.433 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.433 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.433 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:27:19.433 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_first_committed": 1, 2024-08-18T21:27:19.433 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_last_committed": 97, 2024-08-18T21:27:19.433 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_first_committed": 1, 2024-08-18T21:27:19.434 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_last_committed": 392, 2024-08-18T21:27:19.434 INFO:tasks.workunit.client.0.smithi099.stdout: "paxos": { 2024-08-18T21:27:19.434 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:27:19.434 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 537, 2024-08-18T21:27:19.434 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pn": 0, 2024-08-18T21:27:19.434 INFO:tasks.workunit.client.0.smithi099.stdout: "accepted_pn": 0 2024-08-18T21:27:19.434 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:27:19.434 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:27:19.435 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-18T21:27:19.435 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-08-18T21:27:19.435 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-08-18T21:27:19.435 INFO:tasks.workunit.client.0.smithi099.stdout:writing non-divergent object existing_4 2024-08-18T21:27:19.435 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-08-18T21:27:19.436 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-08-18T21:27:19.765 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:27:19.765 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-08-18T21:26:41.666623+0000 29'100 41:130 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:54:08.588726+0000 0 0 2024-08-18T21:27:19.765 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:27:19.765 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:27:19.766 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:27:19.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-08-18T21:27:19.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-08-18T21:27:19.892 INFO:tasks.workunit.client.0.smithi099.stdout:delay recovery 2024-08-18T21:27:19.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-08-18T21:27:20.293 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:27:20.293 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-08-18T21:26:41.666623+0000 29'100 41:130 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:54:08.588726+0000 0 0 2024-08-18T21:27:20.294 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:27:20.294 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:27:20.294 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:27:20.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-08-18T21:27:20.310 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-08-18T21:27:20.310 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-18T21:27:20.310 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-18T21:27:20.311 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:27:20.311 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:27:20.311 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:27:20.311 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.0.asok 2024-08-18T21:27:20.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-08-18T21:27:20.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.0.asok set_recovery_delay 100000 2024-08-18T21:27:20.421 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:27:20.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-08-18T21:27:20.434 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-08-18T21:27:20.434 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-18T21:27:20.434 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-18T21:27:20.435 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:27:20.435 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:27:20.435 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:27:20.436 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.2.asok 2024-08-18T21:27:20.436 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-08-18T21:27:20.436 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.2.asok set_recovery_delay 100000 2024-08-18T21:27:20.550 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:27:20.562 INFO:tasks.workunit.client.0.smithi099.stdout:revive divergent 1 2024-08-18T21:27:20.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-08-18T21:27:20.563 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-08-18T21:27:20.961 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:27:20.961 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-08-18T21:26:41.666623+0000 29'100 41:130 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:54:08.588726+0000 0 0 2024-08-18T21:27:20.962 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:27:20.962 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:27:20.962 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:27:20.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-08-18T21:27:21.641 INFO:tasks.workunit.client.0.smithi099.stderr:noup is set 2024-08-18T21:27:21.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-08-18T21:27:21.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:27:21.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:27:21.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-18T21:27:21.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:27:21.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-18T21:27:21.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:27:21.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:27:21.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:27:21.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:27:21.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-18T21:27:21.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-18T21:27:21.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:27:21.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:27:21.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:27:21.662 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:27:21.662 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:27:21.662 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:27:21.662 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:27:21.662 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:27:21.662 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:27:21.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-18T21:27:21.664 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:27:21.665 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-18T21:27:21.665 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:27:21.665 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-18T21:27:21.665 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-18T21:27:21.665 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:27:21.665 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:27:21.665 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:27:21.705 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:27:21.705+0000 7f30fdfb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:27:21.708 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:27:21.709+0000 7f30fdfb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:27:21.709 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:27:21.709+0000 7f30fdfb27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:27:22.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-08-18T21:27:22.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-08-18T21:27:22.950 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:27:22.949+0000 7f30fdfb27c0 -1 Falling back to public interface 2024-08-18T21:27:26.139 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:27:26.137+0000 7f30fdfb27c0 -1 osd.1 29 log_to_monitors true 2024-08-18T21:27:27.141 INFO:tasks.workunit.client.0.smithi099.stdout:delay recovery divergent 2024-08-18T21:27:27.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-08-18T21:27:27.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-08-18T21:27:27.555 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:27:27.556 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-18T21:26:41.666623+0000 41'101 43:135 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:54:08.588726+0000 0 0 2024-08-18T21:27:27.556 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:27:27.556 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:27:27.556 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:27:27.571 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-08-18T21:27:27.571 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-18T21:27:27.571 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-18T21:27:27.572 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:27:27.572 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:27:27.572 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:27:27.572 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.1.asok 2024-08-18T21:27:27.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-08-18T21:27:27.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.1.asok set_recovery_delay 100000 2024-08-18T21:27:27.682 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:27:27.694 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-08-18T21:27:28.504 INFO:tasks.workunit.client.0.smithi099.stderr:noup is unset 2024-08-18T21:27:28.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-08-18T21:27:28.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:27:28.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:27:28.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:27:28.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:27:28.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:27:28.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:27:28.529 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:27:28.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:27:28.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:27:29.139 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/2203422713,v1:127.0.0.1:6803/2203422713] [v2:127.0.0.1:6804/2203422713,v1:127.0.0.1:6805/2203422713] exists,up 81ca0476-b29b-4284-91e8-459871ca9868 2024-08-18T21:27:29.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:27:29.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:27:29.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:27:29.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-08-18T21:27:29.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:27:29.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:27:29.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:27:29.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:27:29.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:27:29.141 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:27:29.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:27:29.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:27:29.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:27:29.579 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2606823994,v1:127.0.0.1:6819/2606823994] [v2:127.0.0.1:6820/2606823994,v1:127.0.0.1:6821/2606823994] exists,up 57cc8ad2-e869-467a-9640-a317277ee238 2024-08-18T21:27:29.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:27:29.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:27:29.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:27:29.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-08-18T21:27:29.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:27:29.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:27:29.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:27:29.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:27:29.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:27:29.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:27:29.613 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:27:29.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:27:29.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:27:30.004 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2422497712,v1:127.0.0.1:6811/2422497712] [v2:127.0.0.1:6812/2422497712,v1:127.0.0.1:6813/2422497712] exists,up e59c966a-f886-4300-b1de-6e9155227bbc 2024-08-18T21:27:30.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:27:30.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:27:30.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:27:30.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-08-18T21:27:30.431 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:27:30.431 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-18T21:26:41.666623+0000 41'101 43:135 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:54:08.588726+0000 0 0 2024-08-18T21:27:30.431 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:27:30.431 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:27:30.431 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:27:30.447 INFO:tasks.workunit.client.0.smithi099.stdout:wait for peering 2024-08-18T21:27:30.447 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-08-18T21:27:30.447 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-08-18T21:27:30.848 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:27:30.848 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-18T21:26:41.666623+0000 41'101 43:135 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:54:08.588726+0000 0 0 2024-08-18T21:27:30.848 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:27:30.849 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:27:30.849 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:27:30.863 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-08-18T21:27:31.017 INFO:tasks.workunit.client.0.smithi099.stdout:killing divergent 1 2024-08-18T21:27:31.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-08-18T21:27:31.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-08-18T21:27:31.431 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:27:31.431 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-18T21:26:41.666623+0000 41'101 43:135 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:54:08.588726+0000 0 0 2024-08-18T21:27:31.431 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:27:31.431 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:27:31.432 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:27:31.447 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-08-18T21:27:31.448 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:27:31.448 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:27:31.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:27:31.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:27:31.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:27:31.555 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:27:31.555 INFO:tasks.workunit.client.0.smithi099.stdout:reviving divergent 1 2024-08-18T21:27:31.556 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-08-18T21:27:31.556 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-08-18T21:27:31.951 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:27:31.951 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 1 1 0 0 62667 0 0 18 84 18 active+recovering+degraded 2024-08-18T21:27:29.858274+0000 47'102 47:145 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 no scrub is scheduled 0 0 2024-08-18T21:27:31.951 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:27:31.951 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:27:31.951 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:27:31.969 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-08-18T21:27:31.969 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:27:31.969 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:27:31.969 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-18T21:27:31.969 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:27:31.970 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:27:31.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:27:31.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:27:31.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-18T21:27:31.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-18T21:27:31.972 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:27:31.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:27:31.973 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-18T21:27:31.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-18T21:27:31.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:27:31.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:27:31.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:27:32.003 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:27:32.001+0000 7f10fa9b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:27:32.024 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:27:32.025+0000 7f10fa9b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:27:32.025 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:27:32.025+0000 7f10fa9b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:27:32.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-18T21:27:32.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:27:32.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:27:32.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:27:32.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:27:32.472 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:27:32.473 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:27:32.473 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:27:32.473 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:27:32.473 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:27:32.892 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2606823994,v1:127.0.0.1:6819/2606823994] [v2:127.0.0.1:6820/2606823994,v1:127.0.0.1:6821/2606823994] exists,up 57cc8ad2-e869-467a-9640-a317277ee238 2024-08-18T21:27:32.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:27:32.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:27:32.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:27:32.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-08-18T21:27:33.098 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:27:33.097+0000 7f10fa9b37c0 -1 Falling back to public interface 2024-08-18T21:27:36.228 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:27:36.229+0000 7f10fa9b37c0 -1 osd.1 47 log_to_monitors true 2024-08-18T21:27:52.894 INFO:tasks.workunit.client.0.smithi099.stdout:allowing recovery 2024-08-18T21:27:52.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-08-18T21:27:52.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-08-18T21:27:53.315 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:27:53.315 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-08-18T21:27:38.219222+0000 47'102 52:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-20T09:06:35.374382+0000 0 0 2024-08-18T21:27:53.315 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:27:53.316 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:27:53.316 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:27:53.331 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-08-18T21:27:53.331 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-08-18T21:27:53.480 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:27:53.493 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-08-18T21:27:53.494 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-08-18T21:27:53.631 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:27:53.644 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-08-18T21:27:53.644 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-08-18T21:27:53.785 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:27:53.798 INFO:tasks.workunit.client.0.smithi099.stdout:reading divergent objects 2024-08-18T21:27:53.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-08-18T21:27:53.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-08-18T21:27:54.261 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:27:54.261 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-08-18T21:27:38.219222+0000 47'102 52:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:25:38.017873+0000 0'0 2024-08-18T21:25:38.017873+0000 0 0 periodic scrub scheduled @ 2024-08-20T09:06:35.374382+0000 0 0 2024-08-18T21:27:54.261 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:27:54.262 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:27:54.262 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:27:54.279 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-08-18T21:27:54.280 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-08-18T21:27:54.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:27:54.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-08-18T21:27:54.359 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:27:54.359 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-08-18T21:27:54.394 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:27:54.395 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-08-18T21:27:54.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:27:54.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-08-18T21:27:54.465 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-08-18T21:27:54.466 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-08-18T21:27:54.467 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-18T21:27:54.473 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:27:29.017+0000 7f30e1f3c640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-08-18T21:26:13.957277+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-18T21:27:54.473 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:27:29.017+0000 7f30e1f3c640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-08-18T21:26:13.957277+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-18T21:27:54.473 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:27:29.017+0000 7f30e1f3c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:27:54.473 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:27:29.017+0000 7f30e1f3c640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-08-18T21:27:54.473 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:27:29.017+0000 7f30e1f3c640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-08-18T21:27:54.473 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:27:29.017+0000 7f30e1f3c640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-08-18T21:27:54.473 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:27:29.017+0000 7f30e1f3c640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-08-18T21:27:54.473 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:27:29.017+0000 7f30e1f3c640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-08-18T21:26:13.957277+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-18T21:27:54.473 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:27:29.017+0000 7f30e1f3c640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-08-18T21:27:54.474 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-08-18T21:27:54.474 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-18T21:27:54.478 INFO:tasks.workunit.client.0.smithi099.stdout:success 2024-08-18T21:27:54.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-08-18T21:27:54.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-08-18T21:27:54.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-08-18T21:27:54.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-08-18T21:27:55.002 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'test' does not exist 2024-08-18T21:27:55.018 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-08-18T21:27:55.018 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:27:55.018 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:27:55.019 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:27:55.019 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:27:55.019 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:28:00.335 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:28:00.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-08-18T21:28:00.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-18T21:28:00.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:28:00.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-18T21:28:00.337 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:28:00.337 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:28:00.337 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:28:00.337 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:28:00.337 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:28:00.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:28:00.345 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:28:00.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:28:00.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:28:00.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:28:00.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:28:00.347 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:28:00.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:28:00.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:28:00.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:28:00.348 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:28:00.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:28:00.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:28:00.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-18T21:28:00.373 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:28:00.373 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:28:00.373 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:28:00.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27423 2024-08-18T21:28:00.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:28:00.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:28:00.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-08-18T21:28:00.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-08-18T21:28:00.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-08-18T21:28:00.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-08-18T21:28:00.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-18T21:28:00.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:28:00.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-18T21:28:00.376 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:28:00.376 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:28:00.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:28:00.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:28:00.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:28:00.377 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:28:00.377 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:28:00.378 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:28:00.378 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:28:00.379 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:28:00.379 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:28:00.380 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:28:00.380 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:28:00.380 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:28:00.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:28:00.381 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:28:00.382 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:28:00.383 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:28:00.383 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-18T21:28:00.384 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:28:00.384 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:28:00.384 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:28:00.384 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27423 2024-08-18T21:28:00.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:28:00.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:28:00.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-08-18T21:28:00.386 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-18T21:28:00.386 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:28:00.387 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:28:00.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27423 2024-08-18T21:28:00.388 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-18T21:28:00.388 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-18T21:28:00.388 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-18T21:28:00.388 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-08-18T21:28:00.389 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-08-18T21:28:00.389 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-08-18T21:28:00.389 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-08-18T21:28:00.389 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-08-18T21:28:00.389 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-08-18T21:28:00.389 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-08-18T21:28:00.389 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-08-18T21:28:00.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-08-18T21:28:00.391 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:28:00.391 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:28:00.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-08-18T21:28:00.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-08-18T21:28:00.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-18T21:28:00.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-18T21:28:00.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-18T21:28:00.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-08-18T21:28:00.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-08-18T21:28:00.846 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-18T21:28:00.847 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:28:00.847 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:28:00.847 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:28:00.847 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:28:00.847 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:28:00.848 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:28:00.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-18T21:28:01.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-18T21:28:01.068 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-18T21:28:01.068 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:28:01.069 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:28:01.069 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-18T21:28:01.069 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-18T21:28:01.070 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:28:01.070 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:28:01.070 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:28:01.070 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:28:01.070 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:28:01.070 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:28:01.071 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-mon.a.asok 2024-08-18T21:28:01.071 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:28:01.071 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27423/ceph-mon.a.asok config get fsid 2024-08-18T21:28:01.188 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-18T21:28:01.188 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:28:01.188 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:28:01.188 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-18T21:28:01.189 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-18T21:28:01.189 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:28:01.189 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:28:01.189 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:28:01.189 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:28:01.189 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:28:01.189 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:28:01.190 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-mon.a.asok 2024-08-18T21:28:01.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:28:01.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27423/ceph-mon.a.asok config get mon_host 2024-08-18T21:28:01.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-08-18T21:28:01.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-08-18T21:28:01.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-18T21:28:01.303 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-18T21:28:01.303 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-18T21:28:01.303 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-08-18T21:28:01.303 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-18T21:28:02.230 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-18T21:28:02.230 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:28:02.230 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:28:02.230 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:28:02.230 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:28:02.230 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:28:02.230 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:28:02.231 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:28:02.231 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:28:02.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-18T21:28:02.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:28:02.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:28:02.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:28:02.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:28:02.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:28:02.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:28:02.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:28:02.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:28:02.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:28:02.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:28:02.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:28:02.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-08-18T21:28:02.311 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:28:02.311 INFO:tasks.workunit.client.0.smithi099.stdout:add osd0 81e99168-52c5-4e10-803c-8f245b8f8a04 2024-08-18T21:28:02.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=81e99168-52c5-4e10-803c-8f245b8f8a04 2024-08-18T21:28:02.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 81e99168-52c5-4e10-803c-8f245b8f8a04' 2024-08-18T21:28:02.312 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:28:02.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiZ8JmNpH2ExAADD+SLp8VM424dhUf3XZgrg== 2024-08-18T21:28:02.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiZ8JmNpH2ExAADD+SLp8VM424dhUf3XZgrg=="}' 2024-08-18T21:28:02.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 81e99168-52c5-4e10-803c-8f245b8f8a04 -i td/divergent-priors/0/new.json 2024-08-18T21:28:02.670 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:28:02.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-08-18T21:28:02.686 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiZ8JmNpH2ExAADD+SLp8VM424dhUf3XZgrg== --osd-uuid 81e99168-52c5-4e10-803c-8f245b8f8a04 2024-08-18T21:28:02.728 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:02.728+0000 7f252929a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:02.738 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:02.736+0000 7f252929a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:02.739 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:02.736+0000 7f252929a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:02.739 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:02.740+0000 7f252929a7c0 -1 bdev(0x55d529df9c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-08-18T21:28:02.739 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:02.740+0000 7f252929a7c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-08-18T21:28:09.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-08-18T21:28:09.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:28:09.152 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd0 key to auth repository 2024-08-18T21:28:09.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-18T21:28:09.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:28:09.737 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:28:09.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-18T21:28:09.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:28:09.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:28:09.739 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:28:09.739 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:28:09.778 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:09.776+0000 7f0d6f2fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:09.803 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:09.804+0000 7f0d6f2fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:09.805 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:09.804+0000 7f0d6f2fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:10.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-18T21:28:10.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:28:10.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:28:10.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:28:10.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:28:10.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:10.190 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:28:10.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:28:10.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:10.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:10.615 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:10.838 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:10.836+0000 7f0d6f2fc7c0 -1 Falling back to public interface 2024-08-18T21:28:11.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:11.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:11.617 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:28:11.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:28:11.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:11.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:12.100 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:13.102 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:13.102 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:13.103 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:28:13.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:28:13.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:13.104 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:13.472 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:13.472+0000 7f0d6f2fc7c0 -1 osd.0 0 log_to_monitors true 2024-08-18T21:28:13.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:14.538 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:14.538 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:14.538 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:28:14.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:28:14.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:14.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:15.006 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:16.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:16.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:16.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:28:16.009 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:28:16.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:16.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:16.544 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:17.546 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:17.546 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:17.546 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:28:17.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:28:17.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:17.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:17.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:18.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:18.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:18.999 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:28:18.999 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:28:18.999 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:18.999 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:19.626 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:20.627 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:20.627 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:20.628 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:28:20.628 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:28:20.628 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:20.628 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:21.083 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:22.085 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:22.085 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:22.086 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:28:22.086 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:28:22.086 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:22.086 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:22.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:23.559 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:23.559 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:23.559 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:28:23.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:28:23.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:23.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:23.995 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:24.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:24.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:24.998 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:28:24.999 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:28:24.999 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:24.999 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:25.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:26.537 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:26.538 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:26.538 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:28:26.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:28:26.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:26.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:27.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:28.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:28.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:28.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:28:28.046 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:28:28.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:28.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:28.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:29.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:29.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:29.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:28:29.917 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:28:29.917 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:29.917 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:31.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:32.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:32.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:32.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:28:32.046 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:28:32.047 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:32.047 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:32.968 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:33.969 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:33.969 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:33.970 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:28:33.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:28:33.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:33.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:34.789 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:35.791 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:35.791 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:35.791 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:28:35.792 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:28:35.792 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:35.792 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:36.537 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:37.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:37.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:37.540 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:28:37.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:28:37.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:37.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:38.435 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:39.436 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:39.436 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:39.437 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:28:39.437 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:28:39.437 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:39.437 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:28:39.879 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/828655858,v1:127.0.0.1:6803/828655858] [v2:127.0.0.1:6804/828655858,v1:127.0.0.1:6805/828655858] exists,up 81e99168-52c5-4e10-803c-8f245b8f8a04 2024-08-18T21:28:39.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:28:39.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:28:39.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:28:39.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-08-18T21:28:39.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-08-18T21:28:39.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:28:39.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:28:39.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-18T21:28:39.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:28:39.885 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:28:39.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:28:39.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:28:39.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:28:39.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:28:39.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:28:39.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-08-18T21:28:39.887 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:28:39.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=44b90f3c-ec32-432f-9c24-f78d2330efa6 2024-08-18T21:28:39.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 44b90f3c-ec32-432f-9c24-f78d2330efa6' 2024-08-18T21:28:39.888 INFO:tasks.workunit.client.0.smithi099.stdout:add osd1 44b90f3c-ec32-432f-9c24-f78d2330efa6 2024-08-18T21:28:39.888 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:28:39.908 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHZ8Jmp8M6NhAAz3EEPN0Q+1sygpwkdvsY1Q== 2024-08-18T21:28:39.908 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHZ8Jmp8M6NhAAz3EEPN0Q+1sygpwkdvsY1Q=="}' 2024-08-18T21:28:39.908 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 44b90f3c-ec32-432f-9c24-f78d2330efa6 -i td/divergent-priors/1/new.json 2024-08-18T21:28:40.411 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:28:40.426 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-08-18T21:28:40.427 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHZ8Jmp8M6NhAAz3EEPN0Q+1sygpwkdvsY1Q== --osd-uuid 44b90f3c-ec32-432f-9c24-f78d2330efa6 2024-08-18T21:28:40.470 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:40.468+0000 7fe5425ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:40.484 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:40.484+0000 7fe5425ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:40.485 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:40.484+0000 7fe5425ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:40.486 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:40.484+0000 7fe5425ba7c0 -1 bdev(0x55dd95317c00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-08-18T21:28:40.486 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:40.484+0000 7fe5425ba7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-08-18T21:28:46.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-08-18T21:28:46.860 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:28:46.860 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd1 key to auth repository 2024-08-18T21:28:46.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-18T21:28:46.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:28:47.450 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:28:47.450 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-18T21:28:47.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:28:47.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:28:47.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:28:47.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:28:47.491 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:47.492+0000 7f3ba5b097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:47.523 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:47.524+0000 7f3ba5b097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:47.524 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:47.524+0000 7f3ba5b097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:28:47.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-18T21:28:47.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:28:47.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:28:47.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:28:47.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:28:47.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:47.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:28:47.887 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:28:47.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:47.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:28:48.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:48.798 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:48.796+0000 7f3ba5b097c0 -1 Falling back to public interface 2024-08-18T21:28:49.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:49.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:49.365 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:28:49.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:28:49.366 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:49.366 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:28:49.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:50.809 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:50.810 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:50.810 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:28:50.810 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:28:50.811 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:50.811 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:28:51.243 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:51.841 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:28:51.840+0000 7f3ba5b097c0 -1 osd.1 0 log_to_monitors true 2024-08-18T21:28:52.244 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:52.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:52.245 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:28:52.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:28:52.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:52.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:28:52.702 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:53.703 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:53.703 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:53.704 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:28:53.704 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:28:53.704 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:53.704 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:28:54.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:55.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:55.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:55.158 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:28:55.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:28:55.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:55.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:28:55.675 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:56.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:56.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:56.677 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:28:56.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:28:56.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:56.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:28:57.109 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:58.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:58.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:58.112 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:28:58.112 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:28:58.112 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:58.112 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:28:58.679 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:28:59.681 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:28:59.681 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:28:59.681 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:28:59.682 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:28:59.682 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:28:59.682 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:29:00.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:01.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:01.133 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:29:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:29:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:01.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:29:01.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:02.850 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:02.850 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:02.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:29:02.851 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:29:02.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:02.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:29:03.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:04.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:04.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:04.364 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:29:04.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:29:04.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:04.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:29:04.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:05.850 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:05.850 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:05.850 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:29:05.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:29:05.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:05.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:29:06.721 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:07.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:07.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:07.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:29:07.724 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:29:07.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:07.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:29:08.945 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:09.947 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:09.947 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:09.948 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:29:09.948 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:29:09.948 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:09.948 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:29:10.741 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:11.743 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:11.743 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:11.743 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:29:11.744 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:29:11.744 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:11.744 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:29:12.548 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:13.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:13.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:13.550 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:29:13.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:29:13.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:13.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:29:14.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:15.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:15.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:15.168 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:29:15.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:29:15.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:15.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:29:16.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:17.042 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:17.042 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:17.042 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:29:17.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:29:17.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:17.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:29:17.518 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1430101756,v1:127.0.0.1:6811/1430101756] [v2:127.0.0.1:6812/1430101756,v1:127.0.0.1:6813/1430101756] exists,up 44b90f3c-ec32-432f-9c24-f78d2330efa6 2024-08-18T21:29:17.520 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:29:17.520 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:29:17.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:29:17.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-08-18T21:29:17.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-08-18T21:29:17.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:29:17.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:29:17.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-18T21:29:17.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:29:17.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-08-18T21:29:17.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:29:17.522 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:29:17.522 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:29:17.522 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:29:17.522 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-18T21:29:17.522 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-18T21:29:17.522 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:29:17.522 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:29:17.522 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:29:17.522 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:29:17.522 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:29:17.523 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:29:17.523 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:29:17.523 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:29:17.523 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:29:17.523 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:29:17.523 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:29:17.523 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:29:17.523 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:29:17.523 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:29:17.523 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:29:17.523 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:29:17.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:29:17.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:29:17.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:29:17.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:29:17.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:29:17.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:29:17.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-08-18T21:29:17.525 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:29:17.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e90c0df1-349f-4f74-a44f-5ebec21406bd 2024-08-18T21:29:17.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e90c0df1-349f-4f74-a44f-5ebec21406bd' 2024-08-18T21:29:17.525 INFO:tasks.workunit.client.0.smithi099.stdout:add osd2 e90c0df1-349f-4f74-a44f-5ebec21406bd 2024-08-18T21:29:17.526 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:29:17.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtZ8JmDUf3IBAAn1tUhB0DGdrMlqFsztI8Fw== 2024-08-18T21:29:17.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtZ8JmDUf3IBAAn1tUhB0DGdrMlqFsztI8Fw=="}' 2024-08-18T21:29:17.552 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e90c0df1-349f-4f74-a44f-5ebec21406bd -i td/divergent-priors/2/new.json 2024-08-18T21:29:18.229 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:29:18.249 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-08-18T21:29:18.251 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtZ8JmDUf3IBAAn1tUhB0DGdrMlqFsztI8Fw== --osd-uuid e90c0df1-349f-4f74-a44f-5ebec21406bd 2024-08-18T21:29:18.294 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:29:18.291+0000 7f4d5a6ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:29:18.296 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:29:18.295+0000 7f4d5a6ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:29:18.297 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:29:18.295+0000 7f4d5a6ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:29:18.298 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:29:18.295+0000 7f4d5a6ea7c0 -1 bdev(0x55a5f0e7dc00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-08-18T21:29:18.298 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:29:18.295+0000 7f4d5a6ea7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-08-18T21:29:25.514 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-08-18T21:29:25.514 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:29:25.514 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd2 key to auth repository 2024-08-18T21:29:25.515 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-18T21:29:25.515 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:29:26.118 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:29:26.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-18T21:29:26.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:29:26.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:29:26.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:29:26.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:29:26.160 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:29:26.159+0000 7fc4e74557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:29:26.175 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:29:26.175+0000 7fc4e74557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:29:26.176 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:29:26.175+0000 7fc4e74557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:29:26.548 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-18T21:29:26.548 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:29:26.549 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:29:26.549 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:29:26.549 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:29:26.549 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:29:26.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:26.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:29:26.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:26.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:27.037 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:27.653 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:29:27.651+0000 7fc4e74557c0 -1 Falling back to public interface 2024-08-18T21:29:28.039 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:28.039 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:28.039 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:29:28.039 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:29:28.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:28.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:28.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:29.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:29.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:29.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:29:29.491 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:29:29.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:29.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:29.949 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:30.560 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:29:30.559+0000 7fc4e74557c0 -1 osd.2 0 log_to_monitors true 2024-08-18T21:29:30.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:30.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:30.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:29:30.951 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:29:30.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:30.952 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:31.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:32.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:32.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:32.483 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:29:32.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:29:32.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:32.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:32.935 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:33.937 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:33.937 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:33.937 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:29:33.938 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:29:33.938 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:33.938 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:34.595 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:35.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:35.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:35.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:29:35.597 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:29:35.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:35.598 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:36.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:37.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:37.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:37.157 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:29:37.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:29:37.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:37.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:37.653 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:38.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:38.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:38.655 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:29:38.656 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:29:38.656 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:38.656 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:39.109 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:40.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:40.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:40.111 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:29:40.112 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:29:40.112 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:40.112 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:40.563 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:41.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:41.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:41.565 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:29:41.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:29:41.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:41.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:42.042 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:43.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:43.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:43.044 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:29:43.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:29:43.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:43.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:43.581 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:44.582 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:44.583 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:44.583 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:29:44.583 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:29:44.583 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:44.584 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:45.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:46.280 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:46.280 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:46.281 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:29:46.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:29:46.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:46.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:47.118 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:48.120 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:48.120 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:48.120 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:29:48.121 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:29:48.121 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:48.121 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:49.233 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:50.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:50.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:50.235 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:29:50.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:29:50.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:50.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:51.280 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:52.282 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:52.282 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:52.283 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:29:52.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:29:52.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:52.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:52.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:53.920 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:53.920 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:53.921 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:29:53.921 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:29:53.921 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:53.921 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:54.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:29:55.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:29:55.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:29:55.909 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:29:55.910 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:29:55.910 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:29:55.910 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:29:56.494 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3397243287,v1:127.0.0.1:6819/3397243287] [v2:127.0.0.1:6820/3397243287,v1:127.0.0.1:6821/3397243287] exists,up e90c0df1-349f-4f74-a44f-5ebec21406bd 2024-08-18T21:29:56.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:29:56.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:29:56.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:29:56.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-08-18T21:29:57.214 INFO:tasks.workunit.client.0.smithi099.stderr:noout is set 2024-08-18T21:29:57.236 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-08-18T21:29:58.119 INFO:tasks.workunit.client.0.smithi099.stderr:noin is set 2024-08-18T21:29:58.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-08-18T21:29:59.015 INFO:tasks.workunit.client.0.smithi099.stderr:nodown is set 2024-08-18T21:29:59.039 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-08-18T21:29:59.039 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-18T21:29:59.835 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'test' already exists 2024-08-18T21:29:59.853 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-18T21:30:00.855 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-08-18T21:30:01.544 INFO:tasks.workunit.client.0.smithi099.stderr:set pool 1 size to 3 2024-08-18T21:30:01.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-08-18T21:30:02.415 INFO:tasks.workunit.client.0.smithi099.stderr:set pool 1 min_size to 2 2024-08-18T21:30:02.438 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-08-18T21:30:02.438 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:30:02.438 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:30:02.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:30:02.867 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:30:02.867 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:30:02.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:30:02.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:30:02.867 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:30:03.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836502 2024-08-18T21:30:03.029 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836502 2024-08-18T21:30:03.029 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836502' 2024-08-18T21:30:03.029 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:30:03.029 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:30:03.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672981 2024-08-18T21:30:03.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672981 2024-08-18T21:30:03.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836502 1-42949672981' 2024-08-18T21:30:03.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:30:03.188 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:30:03.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509451 2024-08-18T21:30:03.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509451 2024-08-18T21:30:03.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836502 1-42949672981 2-64424509451' 2024-08-18T21:30:03.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:30:03.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-08-18T21:30:03.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:30:03.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:30:03.347 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836502 2024-08-18T21:30:03.347 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:30:03.348 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836502 2024-08-18T21:30:03.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836502 2024-08-18T21:30:03.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-08-18T21:30:03.349 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:30:03.793 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836500 -lt 21474836502 2024-08-18T21:30:03.793 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:30:04.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:30:04.794 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:30:05.243 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836502 -lt 21474836502 2024-08-18T21:30:05.243 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:30:05.244 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672981 2024-08-18T21:30:05.244 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:30:05.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:30:05.246 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672981 2024-08-18T21:30:05.246 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:30:05.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672981 2024-08-18T21:30:05.247 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672981 2024-08-18T21:30:05.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-08-18T21:30:05.247 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:30:05.696 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672981 -lt 42949672981 2024-08-18T21:30:05.696 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:30:05.697 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:30:05.697 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509451 2024-08-18T21:30:05.698 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:30:05.699 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509451 2024-08-18T21:30:05.699 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:30:05.700 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509451 2024-08-18T21:30:05.700 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-08-18T21:30:05.700 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509451 2024-08-18T21:30:05.700 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:30:06.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509451 -lt 64424509451 2024-08-18T21:30:06.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-08-18T21:30:06.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:30:06.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:30:06.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:30:06.136 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-18T21:30:06.136 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:30:06.136 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:30:06.136 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:30:06.136 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:30:06.136 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:30:06.238 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-18T21:30:06.238 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:30:06.238 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:30:06.238 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:30:06.238 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:30:06.238 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:30:06.657 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:30:06.658 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:30:06.658 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:30:06.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:30:06.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:30:06.658 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:30:06.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836505 2024-08-18T21:30:06.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836505 2024-08-18T21:30:06.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836505' 2024-08-18T21:30:06.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:30:06.825 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:30:06.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672983 2024-08-18T21:30:06.980 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672983 2024-08-18T21:30:06.980 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836505 1-42949672983' 2024-08-18T21:30:06.980 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:30:06.980 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:30:07.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509454 2024-08-18T21:30:07.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509454 2024-08-18T21:30:07.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509454' 2024-08-18T21:30:07.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:30:07.135 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836505 2024-08-18T21:30:07.136 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:30:07.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:30:07.137 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836505 2024-08-18T21:30:07.137 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:30:07.138 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836505 2024-08-18T21:30:07.138 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-08-18T21:30:07.138 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836505 2024-08-18T21:30:07.139 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:30:07.645 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836505 -lt 21474836505 2024-08-18T21:30:07.646 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:30:07.646 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-08-18T21:30:07.646 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:30:07.647 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:30:07.648 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672983 2024-08-18T21:30:07.648 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:30:07.649 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672983 2024-08-18T21:30:07.649 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-08-18T21:30:07.650 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672983 2024-08-18T21:30:07.650 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:30:08.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672983 -lt 42949672983 2024-08-18T21:30:08.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:30:08.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509454 2024-08-18T21:30:08.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:30:08.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:30:08.135 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509454 2024-08-18T21:30:08.135 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:30:08.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509454 2024-08-18T21:30:08.136 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509454 2024-08-18T21:30:08.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-08-18T21:30:08.136 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:30:08.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509451 -lt 64424509454 2024-08-18T21:30:08.574 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:30:09.575 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:30:09.575 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:30:10.025 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509454 -lt 64424509454 2024-08-18T21:30:10.026 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:30:10.026 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:30:10.026 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:30:10.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:30:10.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:30:10.591 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:30:10.591 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:30:10.591 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:30:10.591 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:30:10.591 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:30:10.591 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:30:11.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-18T21:30:11.014 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:30:11.014 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:30:11.014 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:30:11.558 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-18T21:30:11.558 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:30:11.558 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:30:11.559 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-08-18T21:30:11.559 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-08-18T21:30:11.977 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:30:11.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-08-18T21:30:11.993 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-08-18T21:30:11.993 INFO:tasks.workunit.client.0.smithi099.stdout:primary and soon to be divergent is 1 2024-08-18T21:30:11.993 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:30:12.414 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:30:12.414 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:30:00.644381+0000 0'0 29:22 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:45:16.129302+0000 0 0 2024-08-18T21:30:12.414 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:30:12.414 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:30:12.414 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:30:12.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-08-18T21:30:12.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-08-18T21:30:12.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-08-18T21:30:12.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-08-18T21:30:12.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-08-18T21:30:12.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-08-18T21:30:12.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-08-18T21:30:12.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-08-18T21:30:12.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-08-18T21:30:12.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-08-18T21:30:12.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-08-18T21:30:12.431 INFO:tasks.workunit.client.0.smithi099.stdout:writing initial objects 2024-08-18T21:30:12.432 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-08-18T21:30:12.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:12.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-08-18T21:30:12.656 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:12.657 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-08-18T21:30:12.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:12.849 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-08-18T21:30:13.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:13.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-08-18T21:30:13.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:13.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-08-18T21:30:13.418 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:13.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-08-18T21:30:13.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:13.604 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-08-18T21:30:13.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:13.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-08-18T21:30:13.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:13.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-08-18T21:30:14.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:14.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-08-18T21:30:14.324 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:14.324 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-08-18T21:30:14.516 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:14.516 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-08-18T21:30:14.701 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:14.701 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-08-18T21:30:14.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:14.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-08-18T21:30:15.077 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:15.078 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-08-18T21:30:15.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:15.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-08-18T21:30:15.420 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:15.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-08-18T21:30:15.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:15.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-08-18T21:30:15.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:15.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-08-18T21:30:15.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:15.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-08-18T21:30:16.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:16.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-08-18T21:30:16.351 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:16.351 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-08-18T21:30:16.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:16.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-08-18T21:30:16.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:16.728 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-08-18T21:30:16.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:16.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-08-18T21:30:17.073 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:17.073 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-08-18T21:30:17.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:17.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-08-18T21:30:17.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:17.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-08-18T21:30:17.659 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:17.659 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-08-18T21:30:17.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:17.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-08-18T21:30:18.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:18.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-08-18T21:30:18.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:18.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-08-18T21:30:18.394 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:18.395 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-08-18T21:30:18.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:18.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-08-18T21:30:18.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:18.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-08-18T21:30:18.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:18.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-08-18T21:30:19.090 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:19.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-08-18T21:30:19.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:19.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-08-18T21:30:19.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:19.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-08-18T21:30:19.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:19.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-08-18T21:30:19.877 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:19.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-08-18T21:30:20.037 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:20.037 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-08-18T21:30:20.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:20.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-08-18T21:30:20.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:20.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-08-18T21:30:20.583 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:20.583 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-08-18T21:30:20.775 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:20.776 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-08-18T21:30:20.968 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:20.968 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-08-18T21:30:21.161 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:21.161 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-08-18T21:30:21.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:21.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-08-18T21:30:21.515 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:21.516 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-08-18T21:30:21.679 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:21.679 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-08-18T21:30:21.847 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:21.847 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-08-18T21:30:22.032 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:22.032 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-08-18T21:30:22.225 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:22.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-08-18T21:30:22.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:22.417 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-08-18T21:30:22.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:22.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-08-18T21:30:22.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:22.827 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-08-18T21:30:23.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:23.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-08-18T21:30:23.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-08-18T21:30:23.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:23.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-08-18T21:30:23.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:23.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-08-18T21:30:23.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:23.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-08-18T21:30:23.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:23.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-08-18T21:30:24.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:24.100 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-08-18T21:30:24.291 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:24.292 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-08-18T21:30:24.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:24.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-08-18T21:30:24.668 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:24.668 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-08-18T21:30:24.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:24.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-08-18T21:30:25.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:25.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-08-18T21:30:25.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:25.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-08-18T21:30:25.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:25.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-08-18T21:30:25.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:25.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-08-18T21:30:25.782 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:25.782 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-08-18T21:30:25.975 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:25.975 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-08-18T21:30:26.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:26.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-08-18T21:30:26.343 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:26.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-08-18T21:30:26.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:26.520 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-08-18T21:30:26.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:26.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-08-18T21:30:26.862 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:26.862 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-08-18T21:30:27.021 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:27.022 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-08-18T21:30:27.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:27.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-08-18T21:30:27.390 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:27.390 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-08-18T21:30:27.574 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:27.574 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-08-18T21:30:27.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:27.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-08-18T21:30:27.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:27.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-08-18T21:30:28.126 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:28.127 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-08-18T21:30:28.303 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:28.303 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-08-18T21:30:28.470 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:28.471 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-08-18T21:30:28.637 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:28.637 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-08-18T21:30:28.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:28.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-08-18T21:30:28.989 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:28.989 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-08-18T21:30:29.232 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:29.232 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-08-18T21:30:29.424 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:29.425 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-08-18T21:30:29.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:29.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-08-18T21:30:29.840 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:29.840 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-08-18T21:30:30.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:30.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-08-18T21:30:30.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:30.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-08-18T21:30:30.337 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:30.338 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-08-18T21:30:30.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:30.505 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-08-18T21:30:30.680 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-08-18T21:30:30.680 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-08-18T21:30:30.865 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-18T21:30:30.865 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-08-18T21:30:30.865 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:30:30.865 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:30:30.866 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:30:30.866 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-18T21:30:30.866 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:30:30.866 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:30:30.867 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:30:30.867 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:30:30.867 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:30:30.941 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-18T21:30:30.941 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:30:30.941 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:30:30.941 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:30:30.941 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:30:30.942 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:30:31.382 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:30:31.382 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:30:31.382 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:30:31.382 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:30:31.382 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:30:31.383 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:30:31.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836512 2024-08-18T21:30:31.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836512 2024-08-18T21:30:31.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836512' 2024-08-18T21:30:31.552 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:30:31.552 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:30:31.700 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672990 2024-08-18T21:30:31.700 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672990 2024-08-18T21:30:31.700 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836512 1-42949672990' 2024-08-18T21:30:31.700 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:30:31.700 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:30:31.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509460 2024-08-18T21:30:31.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509460 2024-08-18T21:30:31.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836512 1-42949672990 2-64424509460' 2024-08-18T21:30:31.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:30:31.861 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836512 2024-08-18T21:30:31.861 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:30:31.862 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:30:31.863 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836512 2024-08-18T21:30:31.863 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:30:31.864 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836512 2024-08-18T21:30:31.864 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836512 2024-08-18T21:30:31.864 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-08-18T21:30:31.864 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:30:32.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836510 -lt 21474836512 2024-08-18T21:30:32.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:30:33.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:30:33.345 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:30:33.837 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836512 -lt 21474836512 2024-08-18T21:30:33.837 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:30:33.837 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672990 2024-08-18T21:30:33.838 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:30:33.839 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:30:33.841 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672990 2024-08-18T21:30:33.841 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:30:33.842 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672990 2024-08-18T21:30:33.843 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672990 2024-08-18T21:30:33.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672990' 2024-08-18T21:30:33.843 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:30:34.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672991 -lt 42949672990 2024-08-18T21:30:34.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:30:34.350 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:30:34.351 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509460 2024-08-18T21:30:34.352 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:30:34.352 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509460 2024-08-18T21:30:34.353 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:30:34.354 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509460 2024-08-18T21:30:34.354 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-08-18T21:30:34.354 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509460 2024-08-18T21:30:34.354 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:30:34.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509461 -lt 64424509460 2024-08-18T21:30:34.803 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:30:34.803 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:30:34.803 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:30:35.383 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:30:35.383 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:30:35.384 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:30:35.384 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:30:35.384 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:30:35.384 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:30:35.384 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:30:35.384 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:30:35.815 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-18T21:30:35.815 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:30:35.816 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:30:35.816 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:30:36.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-18T21:30:36.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:30:36.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:30:36.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-08-18T21:30:36.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-08-18T21:30:36.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-08-18T21:30:36.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-08-18T21:30:36.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-18T21:30:36.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-08-18T21:30:36.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-08-18T21:30:36.803 INFO:tasks.workunit.client.0.smithi099.stdout:blackholing osds 0 2 2024-08-18T21:30:36.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:30:37.226 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:30:37.226 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-08-18T21:30:00.644381+0000 29'100 29:122 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:45:16.129302+0000 0 0 2024-08-18T21:30:37.226 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:30:37.226 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:30:37.226 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:30:37.243 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-08-18T21:30:37.243 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-08-18T21:30:37.243 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-18T21:30:37.243 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-18T21:30:37.244 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:30:37.244 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:30:37.244 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:30:37.244 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.0.asok 2024-08-18T21:30:37.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-08-18T21:30:37.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.0.asok config set objectstore_blackhole 1 2024-08-18T21:30:37.350 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:30:37.351 INFO:tasks.workunit.client.0.smithi099.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-18T21:30:37.351 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:30:37.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-08-18T21:30:37.363 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-08-18T21:30:37.363 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-18T21:30:37.363 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-18T21:30:37.364 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:30:37.364 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:30:37.364 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:30:37.364 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.2.asok 2024-08-18T21:30:37.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-08-18T21:30:37.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.2.asok config set objectstore_blackhole 1 2024-08-18T21:30:37.479 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:30:37.480 INFO:tasks.workunit.client.0.smithi099.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-18T21:30:37.480 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:30:37.492 INFO:tasks.workunit.client.0.smithi099.stdout:create new divergent objects 2024-08-18T21:30:37.492 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-08-18T21:30:37.493 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-08-18T21:30:37.494 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-18T21:30:37.494 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-08-18T21:30:37.494 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-18T21:30:37.495 INFO:tasks.workunit.client.0.smithi099.stdout:writing divergent objects 2024-08-18T21:30:37.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-08-18T21:30:37.496 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-08-18T21:30:37.496 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-08-18T21:30:37.496 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-08-18T21:30:37.497 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-08-18T21:30:37.497 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-08-18T21:30:37.497 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-08-18T21:30:37.497 INFO:tasks.workunit.client.0.smithi099.stdout:remove divergent objects 2024-08-18T21:30:37.498 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-08-18T21:30:37.498 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-08-18T21:30:37.499 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-08-18T21:30:37.500 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-08-18T21:30:37.501 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-08-18T21:30:37.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-08-18T21:30:37.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-08-18T21:30:37.502 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-08-18T21:30:37.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-08-18T21:30:37.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-08-18T21:30:37.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-08-18T21:30:47.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-08-18T21:30:47.511 INFO:tasks.workunit.client.0.smithi099.stderr:rados: no process found 2024-08-18T21:30:47.511 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-08-18T21:30:47.512 INFO:tasks.workunit.client.0.smithi099.stdout:killing all the osds 2024-08-18T21:30:47.512 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:30:47.925 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:30:47.925 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 active+clean 2024-08-18T21:30:00.644381+0000 29'106 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:45:16.129302+0000 0 0 2024-08-18T21:30:47.926 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:30:47.926 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:30:47.926 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:30:47.941 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-08-18T21:30:47.941 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:30:47.941 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:30:47.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:30:47.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:30:47.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:30:48.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:30:48.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-08-18T21:30:48.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-08-18T21:30:48.662 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already down. 2024-08-18T21:30:48.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-08-18T21:30:48.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-08-18T21:30:49.295 INFO:tasks.workunit.client.0.smithi099.stderr:osd.1 is already down. 2024-08-18T21:30:49.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-08-18T21:30:49.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-08-18T21:30:49.866 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already down. 2024-08-18T21:30:49.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-08-18T21:30:49.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-08-18T21:30:50.509 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already out. 2024-08-18T21:30:50.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-08-18T21:30:50.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-08-18T21:30:51.182 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already out. 2024-08-18T21:30:51.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-08-18T21:30:51.202 INFO:tasks.workunit.client.0.smithi099.stdout:bringing up non_divergent 0 2 2024-08-18T21:30:51.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:30:51.601 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:30:51.601 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 stale+active+clean 2024-08-18T21:30:00.644381+0000 29'106 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:45:16.129302+0000 0 0 2024-08-18T21:30:51.601 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:30:51.601 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:30:51.601 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:30:51.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-08-18T21:30:51.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-08-18T21:30:51.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:30:51.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:30:51.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-18T21:30:51.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:30:51.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-18T21:30:51.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:30:51.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:30:51.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:30:51.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:30:51.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-18T21:30:51.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-18T21:30:51.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:30:51.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:30:51.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:30:51.619 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:30:51.619 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:30:51.619 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:30:51.621 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:30:51.621 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:30:51.621 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:30:51.621 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:30:51.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:30:51.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:30:51.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:30:51.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:30:51.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:30:51.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:30:51.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:30:51.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:30:51.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:30:51.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:30:51.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-18T21:30:51.622 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:30:51.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-18T21:30:51.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:30:51.623 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-18T21:30:51.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-18T21:30:51.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:30:51.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:30:51.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:30:51.655 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:30:51.654+0000 7f2f173087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:30:51.657 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:30:51.658+0000 7f2f173087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:30:51.658 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:30:51.658+0000 7f2f173087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:30:52.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-18T21:30:52.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:30:52.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:30:52.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:30:52.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:30:52.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:30:52.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:30:52.071 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:30:52.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:30:52.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:30:52.538 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:30:52.880 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:30:52.878+0000 7f2f173087c0 -1 Falling back to public interface 2024-08-18T21:30:53.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:30:53.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:30:53.540 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:30:53.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:30:53.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:30:53.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:30:53.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:30:54.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:30:54.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:30:54.985 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:30:54.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:30:54.986 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:30:54.986 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:30:55.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:30:56.207 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:30:56.206+0000 7f2f173087c0 -1 osd.0 29 log_to_monitors true 2024-08-18T21:30:56.417 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:30:56.417 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:30:56.417 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:30:56.418 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:30:56.418 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:30:56.418 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:30:56.994 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/602004202,v1:127.0.0.1:6803/602004202] [v2:127.0.0.1:6804/602004202,v1:127.0.0.1:6805/602004202] exists,up 81e99168-52c5-4e10-803c-8f245b8f8a04 2024-08-18T21:30:56.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:30:56.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:30:56.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:30:56.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-08-18T21:30:56.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-08-18T21:30:56.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:30:56.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:30:56.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-18T21:30:56.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:30:56.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-08-18T21:30:56.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:30:56.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:30:56.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:30:56.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:30:56.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-18T21:30:56.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-18T21:30:56.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:30:56.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:30:56.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:30:56.998 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:30:56.998 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:30:56.999 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:30:56.999 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:30:56.999 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:30:56.999 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:30:56.999 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:30:57.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:30:57.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:30:57.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:30:57.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:30:57.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:30:57.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:30:57.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:30:57.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:30:57.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:30:57.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:30:57.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-08-18T21:30:57.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-18T21:30:57.001 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:30:57.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:30:57.002 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-08-18T21:30:57.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-18T21:30:57.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:30:57.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:30:57.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:30:57.039 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:30:57.038+0000 7fe8a2b4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:30:57.062 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:30:57.062+0000 7fe8a2b4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:30:57.063 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:30:57.062+0000 7fe8a2b4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:30:57.460 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-18T21:30:57.460 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:30:57.461 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:30:57.461 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:30:57.461 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:30:57.461 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:30:57.461 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:30:57.461 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:30:57.462 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:30:57.462 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:30:57.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:30:58.356 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:30:58.354+0000 7fe8a2b4c7c0 -1 Falling back to public interface 2024-08-18T21:30:58.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:30:58.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:30:58.889 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:30:58.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:30:58.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:30:58.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:30:59.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:31:00.335 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:31:00.335 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:31:00.335 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:31:00.335 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:31:00.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:31:00.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:31:00.770 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:31:01.771 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:31:01.772 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:31:01.772 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:31:01.772 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:31:01.772 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:31:01.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:31:01.845 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:31:01.846+0000 7fe8a2b4c7c0 -1 osd.2 29 log_to_monitors true 2024-08-18T21:31:02.325 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2213617339,v1:127.0.0.1:6811/2213617339] [v2:127.0.0.1:6812/2213617339,v1:127.0.0.1:6813/2213617339] exists,up e90c0df1-349f-4f74-a44f-5ebec21406bd 2024-08-18T21:31:02.326 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:31:02.326 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:31:02.326 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:31:02.326 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-08-18T21:31:02.327 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-08-18T21:31:02.880 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already in. 2024-08-18T21:31:02.896 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-08-18T21:31:02.897 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-08-18T21:31:03.493 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already in. 2024-08-18T21:31:03.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-18T21:31:03.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-08-18T21:31:03.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:31:03.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:31:03.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:31:03.510 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-18T21:31:03.510 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:31:03.510 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:31:03.510 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:31:03.510 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:31:03.510 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:31:03.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-18T21:31:03.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:31:03.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:31:03.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:31:03.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:31:03.569 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:31:04.059 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:31:04.060 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:31:04.060 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:31:04.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:31:04.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:31:04.060 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:31:04.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=154618822660 2024-08-18T21:31:04.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 154618822660 2024-08-18T21:31:04.219 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-154618822660' 2024-08-18T21:31:04.219 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:31:04.219 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:31:04.331 INFO:tasks.workunit.client.0.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-18T21:31:04.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq= 2024-08-18T21:31:04.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z '' 2024-08-18T21:31:04.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: continue 2024-08-18T21:31:04.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:31:04.334 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:31:04.487 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=163208757251 2024-08-18T21:31:04.488 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 163208757251 2024-08-18T21:31:04.488 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-08-18T21:31:04.488 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:31:04.488 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822660 2024-08-18T21:31:04.488 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:31:04.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:31:04.489 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-154618822660 2024-08-18T21:31:04.489 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:31:04.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=154618822660 2024-08-18T21:31:04.492 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 154618822660 2024-08-18T21:31:04.492 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-08-18T21:31:04.492 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:31:04.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 154618822658 -lt 154618822660 2024-08-18T21:31:04.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:31:05.968 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:31:05.969 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:31:06.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 154618822660 -lt 154618822660 2024-08-18T21:31:06.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:31:06.408 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757251 2024-08-18T21:31:06.408 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:31:06.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:31:06.410 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-163208757251 2024-08-18T21:31:06.410 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:31:06.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=163208757251 2024-08-18T21:31:06.412 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 163208757251 2024-08-18T21:31:06.413 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-08-18T21:31:06.413 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:31:06.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 163208757251 -lt 163208757251 2024-08-18T21:31:06.835 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:31:06.836 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:06.836 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:07.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:31:07.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:07.394 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:07.394 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:07.394 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:07.394 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:07.394 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:07.394 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:07.838 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:07.838 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:07.839 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:07.839 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:08.379 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:08.379 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-18T21:31:08.379 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-18T21:31:08.379 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-18T21:31:08.379 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:08.379 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-18T21:31:08.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:08.480 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:08.481 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:08.481 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:08.481 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:08.481 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:08.481 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:08.481 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:08.899 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:08.900 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:08.900 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:08.900 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:09.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:09.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:09.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:09.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:09.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:09.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:09.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:09.454 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:09.454 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:10.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:10.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:10.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-18T21:31:10.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:10.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-18T21:31:10.216 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:10.216 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:10.217 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:10.217 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:10.218 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:10.218 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:10.218 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:10.218 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:10.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:10.625 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:10.626 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:10.626 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:11.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:11.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:11.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:11.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:11.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:11.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:11.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:11.203 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:11.203 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:11.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:11.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:11.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-18T21:31:11.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:11.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-18T21:31:12.175 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:12.175 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:12.175 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:12.175 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:12.176 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:12.176 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:12.176 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:12.176 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:12.591 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:12.592 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:12.592 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:12.592 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:13.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:13.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:13.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:13.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:13.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:13.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:13.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:13.197 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:13.197 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:13.752 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:13.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:13.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-18T21:31:13.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:13.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-18T21:31:14.554 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:14.554 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:14.555 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:14.555 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:14.555 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:14.555 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:14.556 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:14.556 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:14.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:14.999 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:14.999 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:14.999 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:15.559 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:15.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:15.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:15.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:15.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:15.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:15.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:15.560 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:15.561 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:16.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:16.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:16.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-18T21:31:16.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:16.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-18T21:31:17.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:17.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:17.724 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:17.725 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:17.725 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:17.725 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:17.725 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:17.725 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:18.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:18.151 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:18.152 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:18.152 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:18.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:18.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:18.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:18.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:18.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:18.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:18.717 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:18.717 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:18.717 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:19.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:19.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:19.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-18T21:31:19.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:19.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-18T21:31:22.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:22.508 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:22.509 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:22.509 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:22.509 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:22.509 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:22.509 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:22.509 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:22.949 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:22.950 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:22.950 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:22.950 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:23.529 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:23.529 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:24.083 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:24.084 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:24.084 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-18T21:31:24.084 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:24.084 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-18T21:31:30.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:30.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:30.486 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:30.486 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:30.486 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:30.486 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:30.486 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:30.486 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:30.922 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:30.923 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:30.923 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:30.923 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:31.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:31.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:31.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:31.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:31.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:31.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:31.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:31.529 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:31.529 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:32.107 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:32.107 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:32.108 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-18T21:31:32.108 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:32.108 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-18T21:31:39.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:39.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:39.410 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:39.410 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:39.410 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:39.410 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:39.410 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:39.410 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:39.850 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:39.851 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:39.851 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:39.851 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:40.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:40.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:40.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:40.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:40.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:40.417 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:40.417 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:40.417 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:40.417 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:40.990 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:40.990 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:40.990 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-18T21:31:40.990 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-18T21:31:41.526 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_fingerprint": "73a61311-413c-4542-b8a6-5a5e5e8b0efd", 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "version": "19.0.0-5847-g79766939", 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "commit": "797669399b32d2915e64eb6d4b1257ae3c50f80b", 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "timestamp": "2024-08-18T21:31:41.528681+0000", 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "tag": "", 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "health": { 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "status": "HEALTH_WARN", 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "checks": { 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "OSDMAP_FLAGS": { 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-18T21:31:41.527 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 17 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [], 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: "OSD_DOWN": { 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 osds down", 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:31:41.528 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "osd.1 (root=default,host=smithi099) is down" 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: "PG_DEGRADED": { 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:31:41.529 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:31:41.530 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "application not enabled on pool 'test'" 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: "mutes": [] 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_first_committed": 1, 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_last_committed": 1, 2024-08-18T21:31:41.531 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap": { 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:28:00.420535Z", 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:28:00.420535Z", 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release": 19, 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release_name": "squid", 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "election_strategy": 1, 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "disallowed_leaders: ": "", 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": false, 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "tiebreaker_mon": "", 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_ranks: ": "", 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "features": { 2024-08-18T21:31:41.532 INFO:tasks.workunit.client.0.smithi099.stdout: "persistent": [ 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: "kraken", 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: "luminous", 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: "mimic", 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap-prune", 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: "nautilus", 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: "octopus", 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: "pacific", 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: "elector-pinging", 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: "quincy", 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: "reef", 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: "squid" 2024-08-18T21:31:41.533 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.534 INFO:tasks.workunit.client.0.smithi099.stdout: "optional": [] 2024-08-18T21:31:41.534 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.534 INFO:tasks.workunit.client.0.smithi099.stdout: "mons": [ 2024-08-18T21:31:41.534 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.534 INFO:tasks.workunit.client.0.smithi099.stdout: "rank": 0, 2024-08-18T21:31:41.534 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "a", 2024-08-18T21:31:41.534 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:31:41.534 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.534 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.534 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115", 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 0 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115/0", 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: "priority": 0, 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 0, 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_location": "{}" 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.535 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: "quorum": [ 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: 0 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap": { 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 41, 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:28:01.368432+0000", 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:31:03.616843+0000", 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: "last_up_change": "2024-08-18T21:31:02.149417+0000", 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: "last_in_change": "2024-08-18T21:31:03.263423+0000", 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-18T21:31:41.536 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_num": 5800384, 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_set": [ 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: "nodown", 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: "noin", 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: "noout", 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: "pglog_hardlimit", 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snapdirs", 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: "recovery_deletes", 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: "sortbitwise" 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_version": 9, 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: "full_ratio": 0.99000000953674316, 2024-08-18T21:31:41.537 INFO:tasks.workunit.client.0.smithi099.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_snapshot": "", 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_max": 1, 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: "max_osd": 3, 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: "require_min_compat_client": "luminous", 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: "min_compat_client": "jewel", 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: "require_osd_release": "squid", 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: "allow_crimson": false, 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: "pools": [ 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:31:41.538 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_name": "test", 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "create_time": "2024-08-18T21:29:59.474724+0000", 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": 1, 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_names": "hashpspool", 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "size": 3, 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "min_size": 2, 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_rule": 0, 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "is_stretch_pool": false, 2024-08-18T21:31:41.539 INFO:tasks.workunit.client.0.smithi099.stdout: "object_hash": 2, 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_autoscale_mode": "off", 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num": 1, 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num": 1, 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num_target": 1, 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_target": 1, 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_pending": 1, 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pg_merge_meta": { 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "source_pgid": "0.0", 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "ready_epoch": 0, 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_started": 0, 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": 0, 2024-08-18T21:31:41.540 INFO:tasks.workunit.client.0.smithi099.stdout: "source_version": "0'0", 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "target_version": "0'0" 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "last_change": "29", 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend": "0", 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "auid": 0, 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_mode": "selfmanaged", 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_seq": 0, 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_epoch": 0, 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_snaps": [], 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps": "[]", 2024-08-18T21:31:41.541 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_bytes": 0, 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_objects": 0, 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "tiers": [], 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "tier_of": -1, 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "read_tier": -1, 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "write_tier": -1, 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_mode": "none", 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_bytes": 0, 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_objects": 0, 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-18T21:31:41.542 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_flush_age": 0, 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_evict_age": 0, 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profile": "", 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_params": { 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "none" 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_period": 0, 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_count": 0, 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "use_gmt_hitset": true, 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_search_last_n": 0, 2024-08-18T21:31:41.543 INFO:tasks.workunit.client.0.smithi099.stdout: "grade_table": [], 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "stripe_width": 0, 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "expected_num_objects": 0, 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "fast_read": false, 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "options": {}, 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "application_metadata": {}, 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "read_balance": { 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "score_type": "Fair distribution", 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "score_acting": 2.9900000095367432, 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "score_stable": 2.9900000095367432, 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_score": 0.67000001668930054, 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "raw_score_acting": 2, 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "raw_score_stable": 2, 2024-08-18T21:31:41.544 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity_weighted": 1, 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: "average_primary_affinity": 1, 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: "average_primary_affinity_weighted": 1 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: "osds": [ 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "81e99168-52c5-4e10-803c-8f245b8f8a04", 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:31:41.545 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 5, 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 29, 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 36, 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 40, 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 30, 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6802", 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:31:41.546 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6803", 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6804", 2024-08-18T21:31:41.547 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6805", 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6808", 2024-08-18T21:31:41.548 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6809", 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.549 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.550 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6806", 2024-08-18T21:31:41.550 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:31:41.550 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.550 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.550 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.550 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6807", 2024-08-18T21:31:41.551 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:31:41.551 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.551 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.551 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.551 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6803/602004202", 2024-08-18T21:31:41.551 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6805/602004202", 2024-08-18T21:31:41.551 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6809/602004202", 2024-08-18T21:31:41.551 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6807/602004202", 2024-08-18T21:31:41.551 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:31:41.551 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:31:41.551 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "44b90f3c-ec32-432f-9c24-f78d2330efa6", 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 0, 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 0, 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 0, 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 10, 2024-08-18T21:31:41.552 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 23, 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 31, 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:31:41.553 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:31:41.554 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.555 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.556 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/1430101756", 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/1430101756", 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1430101756", 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1430101756", 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: "exists" 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.557 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "e90c0df1-349f-4f74-a44f-5ebec21406bd", 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 15, 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 29, 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 38, 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 0, 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 32, 2024-08-18T21:31:41.558 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.559 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:31:41.560 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:31:41.561 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:31:41.562 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/2213617339", 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/2213617339", 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2213617339", 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2213617339", 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.563 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_xinfo": [ 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:30:48.420401+0000", 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:28:38.480656+0000", 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:31:41.564 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:30:49.072160+0000", 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:29:16.405791+0000", 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:30:49.696745+0000", 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:31:41.565 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:29:55.837808+0000", 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap": [], 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_items": [], 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_primaries": [], 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_temp": [], 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_temp": [], 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: "blocklist": {}, 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: "range_blocklist": {}, 2024-08-18T21:31:41.566 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profiles": { 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: "default": { 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: "k": "2", 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: "m": "2", 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: "plugin": "jerasure", 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: "technique": "reed_sol_van" 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps_queue": [], 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: "new_removed_snaps": [], 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: "new_purged_snaps": [], 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_node_flags": {}, 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: "device_class_flags": {}, 2024-08-18T21:31:41.567 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": { 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_enabled": false, 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_bucket_count": 0, 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: "degraded_stretch_mode": 0, 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: "recovering_stretch_mode": 0, 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_bucket": 0 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_metadata": [ 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6804/602004202,v1:127.0.0.1:6805/602004202]", 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:31:41.568 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:31:41.569 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:28:08.476930Z", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:31:41.570 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6802/602004202,v1:127.0.0.1:6803/602004202]", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/602004202,v1:127.0.0.1:6809/602004202]", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/602004202,v1:127.0.0.1:6807/602004202]", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:31:41.571 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/0", 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/1430101756,v1:127.0.0.1:6813/1430101756]", 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:31:41.572 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:31:41.573 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:28:46.206013Z", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:31:41.574 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/1430101756,v1:127.0.0.1:6811/1430101756]", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1430101756,v1:127.0.0.1:6817/1430101756]", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1430101756,v1:127.0.0.1:6815/1430101756]", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:31:41.575 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/1", 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/2213617339,v1:127.0.0.1:6813/2213617339]", 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:31:41.576 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:31:41.577 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:29:24.832826Z", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:31:41.578 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/2213617339,v1:127.0.0.1:6811/2213617339]", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2213617339,v1:127.0.0.1:6817/2213617339]", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2213617339,v1:127.0.0.1:6815/2213617339]", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:31:41.579 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/2", 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_clean_epochs": { 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: "min_last_epoch_clean": 0, 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": { 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: "per_pool": [] 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_epochs": [ 2024-08-18T21:31:41.580 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 36 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 38 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_first_committed": 1, 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_last_committed": 41, 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: "crushmap": { 2024-08-18T21:31:41.581 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": [ 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.0", 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.1", 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:31:41.582 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.2", 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: "types": [ 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 0, 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd" 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "host" 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.583 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 2, 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "chassis" 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 3, 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "rack" 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 4, 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "row" 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 5, 2024-08-18T21:31:41.584 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pdu" 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 6, 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pod" 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 7, 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "room" 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 8, 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "datacenter" 2024-08-18T21:31:41.585 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 9, 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "zone" 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 10, 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "region" 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "root" 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.586 INFO:tasks.workunit.client.0.smithi099.stdout: "buckets": [ 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -1, 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default", 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:31:41.587 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -2, 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default~hdd", 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:31:41.588 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099", 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:31:41.589 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.590 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099~hdd", 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:31:41.591 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.592 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: "rules": [ 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_id": 0, 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_name": "replicated_rule", 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: "steps": [ 2024-08-18T21:31:41.593 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "take", 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: "item": -1, 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: "item_name": "default" 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "choose_firstn", 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 0, 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "osd" 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "emit" 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.594 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "tunables": { 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_tries": 0, 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_fallback_tries": 0, 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_total_tries": 50, 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_descend_once": 1, 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_vary_r": 1, 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_stable": 1, 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_descents": 100, 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_collision_tries": 100, 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "straw_calc_version": 1, 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "allowed_bucket_algs": 54, 2024-08-18T21:31:41.595 INFO:tasks.workunit.client.0.smithi099.stdout: "profile": "jewel", 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_tunables": 1, 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "legacy_tunables": 0, 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "minimum_required_version": "jewel", 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables": 1, 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables2": 1, 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v2_rules": 0, 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables3": 1, 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v3_rules": 0, 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v4_buckets": 1, 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables5": 1, 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v5_rules": 0, 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "has_msr_rules": 0 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.596 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_args": {} 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "fsmap": { 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "btime": "2024-08-18T21:28:01:344420+0000", 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "default_fscid": -1, 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": { 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": {}, 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "ro_compat": {}, 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "incompat": { 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_1": "base v0.20", 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_2": "client writeable ranges", 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_3": "default file layouts on dirs", 2024-08-18T21:31:41.597 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_4": "dir inode in separate object", 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_5": "mds uses versioned encoding", 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_8": "no anchor table", 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_9": "file layout v2", 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_10": "snaprealm v2", 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_11": "minor log segments", 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_12": "quiesce subvolumes" 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_flags": { 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: "enable_multiple": true, 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: "ever_enabled_multiple": true 2024-08-18T21:31:41.598 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: "standbys": [], 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: "filesystems": [] 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_first_committed": 1, 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_last_committed": 1, 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: "auth": { 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 5, 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: "num_secrets": 3 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 1, 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_active": 1, 2024-08-18T21:31:41.599 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_unknown": 0, 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osd": 3, 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_sum": { 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 61700, 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 100, 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 0, 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 300, 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 100, 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:31:41.600 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 100, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 100, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 100, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:31:41.601 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:31:41.602 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 0, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 0, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:31:41.603 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 16, 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 16, 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 2, 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 2, 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 0 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum": { 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:31:41.604 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 3, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83660, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2104, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314489140, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036879360, 2024-08-18T21:31:41.605 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2154496, 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 459943, 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10397, 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416931 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:31:41.606 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum_by_class": { 2024-08-18T21:31:41.607 INFO:tasks.workunit.client.0.smithi099.stdout: "hdd": { 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 3, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83660, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2104, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314489140, 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:31:41.608 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036879360, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2154496, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 459943, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10397, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416931 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:31:41.609 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:31:41.610 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_stats": [ 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: "poolid": 1, 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 1, 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 61700, 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 100, 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 0, 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 300, 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:31:41.611 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 100, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 100, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 100, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 100, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:31:41.612 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:31:41.613 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 1220608, 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 184566, 2024-08-18T21:31:41.614 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 16, 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 16, 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 2, 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 2, 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 3 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.615 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_stats": [ 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 154618822667 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 42949672994 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 163208757258 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.616 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_state": [ 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: "state": "active+undersized+degraded", 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 1 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_osd": [ 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.617 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 1, 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 0, 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.618 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": [ 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": {} 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: "servicemap": { 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 5, 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:30:58.987346+0000", 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: "services": { 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": { 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: "daemons": { 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": "", 2024-08-18T21:31:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: "0": { 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: "start_epoch": 0, 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: "start_stamp": "0.000000", 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: "gid": 0, 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "(unrecognized address family 0)/0", 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: "metadata": {}, 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: "task_status": {} 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_first_committed": 1, 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_last_committed": 99, 2024-08-18T21:31:41.620 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_first_committed": 1, 2024-08-18T21:31:41.621 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_last_committed": 385, 2024-08-18T21:31:41.621 INFO:tasks.workunit.client.0.smithi099.stdout: "paxos": { 2024-08-18T21:31:41.621 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:31:41.621 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 532, 2024-08-18T21:31:41.621 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pn": 0, 2024-08-18T21:31:41.621 INFO:tasks.workunit.client.0.smithi099.stdout: "accepted_pn": 0 2024-08-18T21:31:41.621 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:31:41.621 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:31:41.622 INFO:tasks.workunit.client.0.smithi099.stderr:report 1754641419 2024-08-18T21:31:41.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-18T21:31:41.622 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-08-18T21:31:41.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-08-18T21:31:41.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-08-18T21:31:41.622 INFO:tasks.workunit.client.0.smithi099.stdout:writing non-divergent object existing_4 2024-08-18T21:31:41.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:31:41.967 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:31:41.968 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-08-18T21:31:03.900145+0000 29'100 41:131 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T03:43:32.938919+0000 0 0 2024-08-18T21:31:41.968 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:31:41.968 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:31:41.968 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:31:41.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-08-18T21:31:42.108 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-18T21:31:42.108 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-08-18T21:31:42.109 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:31:42.109 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:31:42.109 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:31:42.109 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-18T21:31:42.110 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:31:42.110 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:31:42.110 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:31:42.110 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:31:42.110 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:31:42.183 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-18T21:31:42.183 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:31:42.183 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:31:42.183 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:31:42.183 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:31:42.183 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:31:42.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:31:42.618 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:31:42.618 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:31:42.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:31:42.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:31:42.618 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:31:42.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=154618822670 2024-08-18T21:31:42.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 154618822670 2024-08-18T21:31:42.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-154618822670' 2024-08-18T21:31:42.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:31:42.823 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:31:42.936 INFO:tasks.workunit.client.0.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-18T21:31:42.938 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq= 2024-08-18T21:31:42.938 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z '' 2024-08-18T21:31:42.938 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: continue 2024-08-18T21:31:42.938 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:31:42.938 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:31:43.123 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=163208757261 2024-08-18T21:31:43.123 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 163208757261 2024-08-18T21:31:43.123 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-154618822670 2-163208757261' 2024-08-18T21:31:43.124 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:31:43.124 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822670 2024-08-18T21:31:43.124 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:31:43.125 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:31:43.125 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-154618822670 2024-08-18T21:31:43.126 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:31:43.126 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=154618822670 2024-08-18T21:31:43.127 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 154618822670 2024-08-18T21:31:43.127 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 154618822670' 2024-08-18T21:31:43.127 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:31:43.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 154618822670 -lt 154618822670 2024-08-18T21:31:43.552 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:31:43.552 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:31:43.552 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757261 2024-08-18T21:31:43.553 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:31:43.554 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-163208757261 2024-08-18T21:31:43.554 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:31:43.555 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=163208757261 2024-08-18T21:31:43.556 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 163208757261 2024-08-18T21:31:43.556 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 163208757261' 2024-08-18T21:31:43.556 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:31:43.982 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 163208757259 -lt 163208757261 2024-08-18T21:31:43.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:31:44.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:31:44.985 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:31:45.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 163208757261 -lt 163208757261 2024-08-18T21:31:45.420 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:31:45.420 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:45.420 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:45.991 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:31:45.991 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:45.991 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:45.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:45.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:45.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:45.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:45.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:46.402 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:46.402 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:46.403 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:46.403 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:46.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:46.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-18T21:31:46.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-18T21:31:46.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-18T21:31:46.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:46.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-18T21:31:47.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:47.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:47.044 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:47.044 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:47.044 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:47.044 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:47.045 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:47.045 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:47.466 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:47.467 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:47.467 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:47.467 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:48.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:48.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:48.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:48.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:48.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:48.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:48.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:48.011 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:48.011 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:48.653 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:48.654 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:48.654 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-18T21:31:48.654 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:48.654 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-18T21:31:48.854 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:48.855 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:48.855 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:48.856 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:48.856 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:48.856 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:48.856 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:48.856 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:49.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:49.276 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:49.277 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:49.277 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:49.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:49.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:49.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:49.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:49.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:49.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:49.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:49.842 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:49.842 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:50.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:50.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:50.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-18T21:31:50.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:50.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-18T21:31:50.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:50.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:50.795 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:50.795 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:50.795 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:50.795 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:50.796 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:50.796 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:51.233 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:51.234 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:51.234 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:51.234 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:51.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:51.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:51.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:51.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:51.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:51.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:51.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:51.808 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:51.808 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:52.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:52.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:52.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-18T21:31:52.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:52.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-18T21:31:53.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:53.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:53.210 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:53.210 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:53.210 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:53.211 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:53.211 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:53.211 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:53.653 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:53.654 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:53.654 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:53.654 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:54.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:54.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:54.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:54.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:54.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:54.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:54.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:54.206 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:54.206 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:54.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:54.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:54.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-18T21:31:54.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:54.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-18T21:31:56.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:31:56.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:31:56.369 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:31:56.369 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:31:56.369 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:31:56.369 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:31:56.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:31:56.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:31:56.788 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:31:56.789 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:31:56.789 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:31:56.789 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:31:57.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:31:57.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:31:57.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:31:57.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:31:57.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:31:57.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:31:57.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:31:57.348 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:31:57.348 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:31:57.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:31:57.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:31:57.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-18T21:31:57.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:31:57.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-18T21:32:01.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:32:01.236 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:32:01.237 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:32:01.237 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:32:01.237 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:32:01.237 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:32:01.237 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:32:01.238 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:32:01.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:32:01.614 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:32:01.614 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:32:01.615 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:32:02.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:32:02.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:32:02.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:32:02.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:32:02.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:32:02.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:32:02.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:32:02.153 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:32:02.154 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:32:02.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:32:02.715 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:32:02.715 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-18T21:32:02.715 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:32:02.715 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-18T21:32:09.115 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:32:09.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:32:09.116 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:32:09.117 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:32:09.117 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:32:09.117 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:32:09.117 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:32:09.117 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:32:09.559 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:32:09.559 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:32:09.559 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:32:09.559 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:32:10.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:32:10.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:32:10.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:32:10.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:32:10.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:32:10.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:32:10.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:32:10.117 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:32:10.117 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:32:10.675 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:32:10.675 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:32:10.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-18T21:32:10.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:32:10.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-18T21:32:17.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:32:17.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:32:17.977 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:32:17.978 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:32:17.978 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:32:17.978 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:32:17.978 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:32:17.978 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:32:18.420 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:32:18.420 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:32:18.421 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:32:18.421 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:32:18.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:32:18.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:32:18.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:32:18.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:32:18.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:32:18.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:32:18.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:32:18.978 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:32:18.978 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:32:19.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:32:19.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:32:19.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-18T21:32:19.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-18T21:32:20.148 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:32:20.148 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_fingerprint": "73a61311-413c-4542-b8a6-5a5e5e8b0efd", 2024-08-18T21:32:20.148 INFO:tasks.workunit.client.0.smithi099.stdout: "version": "19.0.0-5847-g79766939", 2024-08-18T21:32:20.148 INFO:tasks.workunit.client.0.smithi099.stdout: "commit": "797669399b32d2915e64eb6d4b1257ae3c50f80b", 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "timestamp": "2024-08-18T21:32:20.151978+0000", 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "tag": "", 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "health": { 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "status": "HEALTH_WARN", 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "checks": { 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "OSDMAP_FLAGS": { 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 17 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [], 2024-08-18T21:32:20.149 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: "OSD_DOWN": { 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 osds down", 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "osd.1 (root=default,host=smithi099) is down" 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.150 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: "PG_DEGRADED": { 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 2 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.0 is stuck undersized for 75s, current state active+undersized+degraded, last acting [0,2]" 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.151 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "application not enabled on pool 'test'" 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.152 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: "mutes": [] 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_first_committed": 1, 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_last_committed": 1, 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap": { 2024-08-18T21:32:20.153 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:28:00.420535Z", 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:28:00.420535Z", 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release": 19, 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release_name": "squid", 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "election_strategy": 1, 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "disallowed_leaders: ": "", 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": false, 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "tiebreaker_mon": "", 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_ranks: ": "", 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "features": { 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "persistent": [ 2024-08-18T21:32:20.154 INFO:tasks.workunit.client.0.smithi099.stdout: "kraken", 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: "luminous", 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: "mimic", 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap-prune", 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: "nautilus", 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: "octopus", 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: "pacific", 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: "elector-pinging", 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: "quincy", 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: "reef", 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: "squid" 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: "optional": [] 2024-08-18T21:32:20.155 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: "mons": [ 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: "rank": 0, 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "a", 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115", 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 0 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.156 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115/0", 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: "priority": 0, 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 0, 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_location": "{}" 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: "quorum": [ 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: 0 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap": { 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 41, 2024-08-18T21:32:20.157 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:28:01.368432+0000", 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:31:03.616843+0000", 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "last_up_change": "2024-08-18T21:31:02.149417+0000", 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "last_in_change": "2024-08-18T21:31:03.263423+0000", 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_num": 5800384, 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_set": [ 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "nodown", 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "noin", 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "noout", 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "pglog_hardlimit", 2024-08-18T21:32:20.158 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snapdirs", 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "recovery_deletes", 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "sortbitwise" 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_version": 9, 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "full_ratio": 0.99000000953674316, 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_snapshot": "", 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_max": 1, 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "max_osd": 3, 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "require_min_compat_client": "luminous", 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "min_compat_client": "jewel", 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "require_osd_release": "squid", 2024-08-18T21:32:20.159 INFO:tasks.workunit.client.0.smithi099.stdout: "allow_crimson": false, 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "pools": [ 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_name": "test", 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "create_time": "2024-08-18T21:29:59.474724+0000", 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": 1, 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_names": "hashpspool", 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "size": 3, 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "min_size": 2, 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_rule": 0, 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-08-18T21:32:20.160 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "is_stretch_pool": false, 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "object_hash": 2, 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_autoscale_mode": "off", 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num": 1, 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num": 1, 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num_target": 1, 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_target": 1, 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_pending": 1, 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pg_merge_meta": { 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "source_pgid": "0.0", 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "ready_epoch": 0, 2024-08-18T21:32:20.161 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_started": 0, 2024-08-18T21:32:20.162 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": 0, 2024-08-18T21:32:20.162 INFO:tasks.workunit.client.0.smithi099.stdout: "source_version": "0'0", 2024-08-18T21:32:20.162 INFO:tasks.workunit.client.0.smithi099.stdout: "target_version": "0'0" 2024-08-18T21:32:20.162 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.162 INFO:tasks.workunit.client.0.smithi099.stdout: "last_change": "29", 2024-08-18T21:32:20.162 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend": "0", 2024-08-18T21:32:20.162 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-18T21:32:20.162 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-08-18T21:32:20.162 INFO:tasks.workunit.client.0.smithi099.stdout: "auid": 0, 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_mode": "selfmanaged", 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_seq": 0, 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_epoch": 0, 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_snaps": [], 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps": "[]", 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_bytes": 0, 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_objects": 0, 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "tiers": [], 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "tier_of": -1, 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "read_tier": -1, 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "write_tier": -1, 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_mode": "none", 2024-08-18T21:32:20.163 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_bytes": 0, 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_objects": 0, 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_flush_age": 0, 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_evict_age": 0, 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profile": "", 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_params": { 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "none" 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_period": 0, 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_count": 0, 2024-08-18T21:32:20.164 INFO:tasks.workunit.client.0.smithi099.stdout: "use_gmt_hitset": true, 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_search_last_n": 0, 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "grade_table": [], 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "stripe_width": 0, 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "expected_num_objects": 0, 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "fast_read": false, 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "options": {}, 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "application_metadata": {}, 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "read_balance": { 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "score_type": "Fair distribution", 2024-08-18T21:32:20.165 INFO:tasks.workunit.client.0.smithi099.stdout: "score_acting": 2.9900000095367432, 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: "score_stable": 2.9900000095367432, 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_score": 0.67000001668930054, 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: "raw_score_acting": 2, 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: "raw_score_stable": 2, 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity_weighted": 1, 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: "average_primary_affinity": 1, 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: "average_primary_affinity_weighted": 1 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: "osds": [ 2024-08-18T21:32:20.166 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "81e99168-52c5-4e10-803c-8f245b8f8a04", 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 5, 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 29, 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 36, 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 40, 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 30, 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:32:20.167 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6802", 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6803", 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.168 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6804", 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6805", 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.169 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6808", 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6809", 2024-08-18T21:32:20.170 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6806", 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6807", 2024-08-18T21:32:20.171 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 602004202 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6803/602004202", 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6805/602004202", 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6809/602004202", 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6807/602004202", 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.172 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "44b90f3c-ec32-432f-9c24-f78d2330efa6", 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 0, 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 0, 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 0, 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 10, 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 23, 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 31, 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:32:20.173 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.174 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.175 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.176 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1430101756 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.177 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/1430101756", 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/1430101756", 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1430101756", 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1430101756", 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: "exists" 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "e90c0df1-349f-4f74-a44f-5ebec21406bd", 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:32:20.178 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 15, 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 29, 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 38, 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 0, 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 32, 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.179 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.180 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:32:20.181 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.182 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2213617339 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.183 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/2213617339", 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/2213617339", 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2213617339", 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2213617339", 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_xinfo": [ 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.184 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:30:48.420401+0000", 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:28:38.480656+0000", 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:30:49.072160+0000", 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:32:20.185 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:29:16.405791+0000", 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:30:49.696745+0000", 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:29:55.837808+0000", 2024-08-18T21:32:20.186 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap": [], 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_items": [], 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_primaries": [], 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_temp": [], 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_temp": [], 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: "blocklist": {}, 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: "range_blocklist": {}, 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profiles": { 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: "default": { 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: "k": "2", 2024-08-18T21:32:20.187 INFO:tasks.workunit.client.0.smithi099.stdout: "m": "2", 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: "plugin": "jerasure", 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: "technique": "reed_sol_van" 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps_queue": [], 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: "new_removed_snaps": [], 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: "new_purged_snaps": [], 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_node_flags": {}, 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: "device_class_flags": {}, 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": { 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_enabled": false, 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_bucket_count": 0, 2024-08-18T21:32:20.188 INFO:tasks.workunit.client.0.smithi099.stdout: "degraded_stretch_mode": 0, 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: "recovering_stretch_mode": 0, 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_bucket": 0 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_metadata": [ 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6804/602004202,v1:127.0.0.1:6805/602004202]", 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:32:20.189 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:32:20.190 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:28:08.476930Z", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:32:20.191 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6802/602004202,v1:127.0.0.1:6803/602004202]", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/602004202,v1:127.0.0.1:6809/602004202]", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/602004202,v1:127.0.0.1:6807/602004202]", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:32:20.192 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/0", 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/1430101756,v1:127.0.0.1:6813/1430101756]", 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:32:20.193 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:32:20.194 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:28:46.206013Z", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:32:20.195 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/1430101756,v1:127.0.0.1:6811/1430101756]", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1430101756,v1:127.0.0.1:6817/1430101756]", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1430101756,v1:127.0.0.1:6815/1430101756]", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:32:20.196 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/1", 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/2213617339,v1:127.0.0.1:6813/2213617339]", 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:32:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:32:20.198 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:29:24.832826Z", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:32:20.199 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/2213617339,v1:127.0.0.1:6811/2213617339]", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2213617339,v1:127.0.0.1:6817/2213617339]", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2213617339,v1:127.0.0.1:6815/2213617339]", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:32:20.200 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/2", 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_clean_epochs": { 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: "min_last_epoch_clean": 0, 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": { 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: "per_pool": [] 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_epochs": [ 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.201 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 36 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 38 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_first_committed": 1, 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_last_committed": 41, 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: "crushmap": { 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": [ 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.202 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.0", 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.1", 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.2", 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:32:20.203 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: "types": [ 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 0, 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd" 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "host" 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 2, 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "chassis" 2024-08-18T21:32:20.204 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 3, 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "rack" 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 4, 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "row" 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 5, 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pdu" 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.205 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 6, 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pod" 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 7, 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "room" 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 8, 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "datacenter" 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 9, 2024-08-18T21:32:20.206 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "zone" 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 10, 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "region" 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "root" 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: "buckets": [ 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.207 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -1, 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default", 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.208 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -2, 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default~hdd", 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:32:20.209 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099", 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:32:20.210 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:32:20.211 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099~hdd", 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.212 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.213 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: "rules": [ 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_id": 0, 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_name": "replicated_rule", 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: "steps": [ 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "take", 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: "item": -1, 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: "item_name": "default" 2024-08-18T21:32:20.214 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "choose_firstn", 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 0, 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "osd" 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "emit" 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: "tunables": { 2024-08-18T21:32:20.215 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_tries": 0, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_fallback_tries": 0, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_total_tries": 50, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_descend_once": 1, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_vary_r": 1, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_stable": 1, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_descents": 100, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_collision_tries": 100, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "straw_calc_version": 1, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "allowed_bucket_algs": 54, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "profile": "jewel", 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_tunables": 1, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "legacy_tunables": 0, 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "minimum_required_version": "jewel", 2024-08-18T21:32:20.216 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables": 1, 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables2": 1, 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v2_rules": 0, 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables3": 1, 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v3_rules": 0, 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v4_buckets": 1, 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables5": 1, 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v5_rules": 0, 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: "has_msr_rules": 0 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_args": {} 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: "fsmap": { 2024-08-18T21:32:20.217 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:32:20.218 INFO:tasks.workunit.client.0.smithi099.stderr:report 4087790033 2024-08-18T21:32:20.218 INFO:tasks.workunit.client.0.smithi099.stdout: "btime": "2024-08-18T21:28:01:344420+0000", 2024-08-18T21:32:20.218 INFO:tasks.workunit.client.0.smithi099.stdout: "default_fscid": -1, 2024-08-18T21:32:20.218 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": { 2024-08-18T21:32:20.218 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": {}, 2024-08-18T21:32:20.218 INFO:tasks.workunit.client.0.smithi099.stdout: "ro_compat": {}, 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "incompat": { 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_1": "base v0.20", 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_2": "client writeable ranges", 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_3": "default file layouts on dirs", 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_4": "dir inode in separate object", 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_5": "mds uses versioned encoding", 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_8": "no anchor table", 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_9": "file layout v2", 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_10": "snaprealm v2", 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_11": "minor log segments", 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_12": "quiesce subvolumes" 2024-08-18T21:32:20.219 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_flags": { 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: "enable_multiple": true, 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: "ever_enabled_multiple": true 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: "standbys": [], 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: "filesystems": [] 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_first_committed": 1, 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_last_committed": 1, 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: "auth": { 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:32:20.220 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 5, 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "num_secrets": 3 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 1, 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_active": 1, 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_unknown": 0, 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osd": 3, 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_sum": { 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 62050, 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 100, 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 0, 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 300, 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:32:20.221 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 100, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 100, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 101, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 101, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:32:20.222 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:32:20.223 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:32:20.224 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 0, 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 0, 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 17, 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 17, 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 2, 2024-08-18T21:32:20.225 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 2, 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 0 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum": { 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 3, 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83660, 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2104, 2024-08-18T21:32:20.226 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314489140, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036879360, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2154496, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 460643, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10397, 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416931 2024-08-18T21:32:20.227 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:32:20.228 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum_by_class": { 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: "hdd": { 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 3, 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:32:20.229 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83660, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2104, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314489140, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036879360, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2154496, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 460643, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:32:20.230 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10397, 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416931 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.231 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_stats": [ 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: "poolid": 1, 2024-08-18T21:32:20.232 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 1, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 62050, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 100, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 0, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 300, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 100, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 100, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:32:20.233 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 101, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 101, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:32:20.234 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:32:20.235 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 1220608, 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 185266, 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:32:20.236 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 17, 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 17, 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 2, 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 2, 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 3 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_stats": [ 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 154618822677 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.237 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 42949672994 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 163208757268 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_state": [ 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: "state": "active+undersized+degraded", 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 1 2024-08-18T21:32:20.238 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_osd": [ 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 1, 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:32:20.239 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 0, 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": [ 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": {} 2024-08-18T21:32:20.240 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: "servicemap": { 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 7, 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:32:03.003639+0000", 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: "services": {} 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_first_committed": 1, 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_last_committed": 118, 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_first_committed": 1, 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_last_committed": 455, 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: "paxos": { 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:32:20.241 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 621, 2024-08-18T21:32:20.242 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pn": 0, 2024-08-18T21:32:20.242 INFO:tasks.workunit.client.0.smithi099.stdout: "accepted_pn": 0 2024-08-18T21:32:20.242 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:32:20.242 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:32:20.242 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-18T21:32:20.242 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-08-18T21:32:20.243 INFO:tasks.workunit.client.0.smithi099.stdout:delay recovery 2024-08-18T21:32:20.243 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:32:20.574 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:32:20.575 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-18T21:31:03.900145+0000 41'101 41:132 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T03:43:32.938919+0000 0 0 2024-08-18T21:32:20.575 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:32:20.575 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:32:20.575 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:32:20.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-08-18T21:32:20.590 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-08-18T21:32:20.590 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-18T21:32:20.590 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-18T21:32:20.591 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:32:20.591 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:32:20.591 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:32:20.591 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.0.asok 2024-08-18T21:32:20.592 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-08-18T21:32:20.592 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.0.asok set_recovery_delay 100000 2024-08-18T21:32:20.697 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:32:20.708 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-08-18T21:32:20.709 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-08-18T21:32:20.709 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-18T21:32:20.709 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-18T21:32:20.709 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:32:20.709 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:32:20.710 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:32:20.710 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.2.asok 2024-08-18T21:32:20.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-08-18T21:32:20.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.2.asok set_recovery_delay 100000 2024-08-18T21:32:20.824 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:32:20.838 INFO:tasks.workunit.client.0.smithi099.stdout:revive divergent 1 2024-08-18T21:32:20.838 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-08-18T21:32:20.838 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:32:21.234 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:32:21.234 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-18T21:31:03.900145+0000 41'101 41:132 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T03:43:32.938919+0000 0 0 2024-08-18T21:32:21.234 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:32:21.235 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:32:21.235 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:32:21.249 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-08-18T21:32:21.947 INFO:tasks.workunit.client.0.smithi099.stderr:noup is set 2024-08-18T21:32:21.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-08-18T21:32:21.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:32:21.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:32:21.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-18T21:32:21.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:32:21.971 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:32:21.973 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:32:21.973 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:32:21.973 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:32:21.973 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:32:21.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:32:21.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:32:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:32:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:32:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:32:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:32:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:32:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:32:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:32:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:32:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-18T21:32:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-18T21:32:21.977 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:32:21.978 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:32:21.978 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-18T21:32:21.978 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-18T21:32:21.978 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:32:21.978 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:32:21.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:32:22.016 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:32:22.013+0000 7fc3da6d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:32:22.039 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:32:22.037+0000 7fc3da6d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:32:22.041 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:32:22.041+0000 7fc3da6d07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:32:22.404 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-08-18T21:32:22.404 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-08-18T21:32:23.103 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:32:23.101+0000 7fc3da6d07c0 -1 Falling back to public interface 2024-08-18T21:32:26.276 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:32:26.277+0000 7fc3da6d07c0 -1 osd.1 29 log_to_monitors true 2024-08-18T21:32:27.405 INFO:tasks.workunit.client.0.smithi099.stdout:delay recovery divergent 2024-08-18T21:32:27.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-08-18T21:32:27.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:32:27.838 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:32:27.838 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-18T21:31:03.900145+0000 41'101 43:136 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T03:43:32.938919+0000 0 0 2024-08-18T21:32:27.838 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:32:27.838 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:32:27.838 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:32:27.855 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-08-18T21:32:27.855 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-18T21:32:27.855 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-18T21:32:27.855 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:32:27.855 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:32:27.856 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:32:27.856 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.1.asok 2024-08-18T21:32:27.856 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-08-18T21:32:27.856 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.1.asok set_recovery_delay 100000 2024-08-18T21:32:27.970 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:32:27.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-08-18T21:32:28.617 INFO:tasks.workunit.client.0.smithi099.stderr:noup is unset 2024-08-18T21:32:28.637 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-08-18T21:32:28.637 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:32:28.637 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:32:28.638 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:32:28.638 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:32:28.638 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:32:28.638 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:32:28.638 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:32:28.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:32:28.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:32:29.210 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/602004202,v1:127.0.0.1:6803/602004202] [v2:127.0.0.1:6804/602004202,v1:127.0.0.1:6805/602004202] exists,up 81e99168-52c5-4e10-803c-8f245b8f8a04 2024-08-18T21:32:29.212 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:32:29.212 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:32:29.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:32:29.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-08-18T21:32:29.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:32:29.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:32:29.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:32:29.214 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:32:29.214 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:32:29.214 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:32:29.214 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:32:29.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:32:29.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:32:29.649 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2968621215,v1:127.0.0.1:6819/2968621215] [v2:127.0.0.1:6820/2968621215,v1:127.0.0.1:6821/2968621215] exists,up 44b90f3c-ec32-432f-9c24-f78d2330efa6 2024-08-18T21:32:29.650 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:32:29.650 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:32:29.650 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:32:29.650 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-08-18T21:32:29.651 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:32:29.651 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:32:29.651 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:32:29.651 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:32:29.651 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:32:29.651 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:32:29.651 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:32:29.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:32:29.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:32:30.126 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/2213617339,v1:127.0.0.1:6811/2213617339] [v2:127.0.0.1:6812/2213617339,v1:127.0.0.1:6813/2213617339] exists,up e90c0df1-349f-4f74-a44f-5ebec21406bd 2024-08-18T21:32:30.127 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:32:30.127 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:32:30.127 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:32:30.127 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:32:30.572 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:32:30.572 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-18T21:31:03.900145+0000 41'101 43:136 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T03:43:32.938919+0000 0 0 2024-08-18T21:32:30.572 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:32:30.572 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:32:30.572 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:32:30.587 INFO:tasks.workunit.client.0.smithi099.stdout:wait for peering 2024-08-18T21:32:30.588 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-08-18T21:32:30.588 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:32:31.002 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:32:31.003 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-18T21:31:03.900145+0000 41'101 43:136 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T03:43:32.938919+0000 0 0 2024-08-18T21:32:31.003 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:32:31.003 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:32:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:32:31.019 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-08-18T21:32:31.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-08-18T21:32:31.186 INFO:tasks.workunit.client.0.smithi099.stdout:killing divergent 1 2024-08-18T21:32:31.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:32:31.599 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:32:31.599 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-08-18T21:31:03.900145+0000 41'101 43:136 [0,2] 0 [0,2] 0 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T03:43:32.938919+0000 0 0 2024-08-18T21:32:31.599 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:32:31.600 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:32:31.600 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:32:31.615 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-08-18T21:32:31.616 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:32:31.616 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:32:31.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:32:31.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:32:31.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:32:31.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:32:31.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.27423.out 2024-08-18T21:32:31.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27423.out 2024-08-18T21:32:31.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-18T21:32:31.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-18T21:32:31.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-18T21:32:31.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-18T21:32:31.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-08-18T21:32:31.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.27423.out 2024-08-18T21:32:33.564 INFO:tasks.workunit.client.0.smithi099.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-08-18T21:32:33.582 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:023dd3d2:::existing_12:head# 2024-08-18T21:32:33.589 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:04c40a0c:::existing_93:head# 2024-08-18T21:32:33.597 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:064ff5fb:::existing_36:head# 2024-08-18T21:32:33.605 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:091f9f1d:::existing_94:head# 2024-08-18T21:32:33.606 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:0a865a93:::existing_54:head# 2024-08-18T21:32:33.606 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:0fed513c:::existing_41:head# 2024-08-18T21:32:33.607 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-08-18T21:32:33.607 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:169157fa:::existing_81:head# 2024-08-18T21:32:33.608 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:17731382:::existing_58:head# 2024-08-18T21:32:33.608 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:18a45a39:::existing_43:head# 2024-08-18T21:32:33.609 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:1c160f01:::existing_66:head# 2024-08-18T21:32:33.609 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:1dd6c249:::existing_97:head# 2024-08-18T21:32:33.610 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:227e0c77:::existing_7:head# 2024-08-18T21:32:33.610 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:23e1b063:::existing_59:head# 2024-08-18T21:32:33.611 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:28566edb:::existing_78:head# 2024-08-18T21:32:33.611 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:2a101182:::existing_20:head# 2024-08-18T21:32:33.612 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:2d1519da:::existing_96:head# 2024-08-18T21:32:33.612 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:3078b87a:::existing_86:head# 2024-08-18T21:32:33.612 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:30bde01b:::existing_15:head# 2024-08-18T21:32:33.613 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:317a7e19:::existing_56:head# 2024-08-18T21:32:33.613 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-08-18T21:32:33.614 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:3e16d84b:::existing_48:head# 2024-08-18T21:32:33.614 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:45fe8203:::existing_75:head# 2024-08-18T21:32:33.615 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-08-18T21:32:33.615 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:5298dcdb:::existing_57:head# 2024-08-18T21:32:33.616 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:52ebaa44:::existing_32:head# 2024-08-18T21:32:33.616 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:53588912:::existing_9:head# 2024-08-18T21:32:33.616 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-08-18T21:32:33.617 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:5da0255c:::existing_13:head# 2024-08-18T21:32:33.617 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:5da2dab8:::existing_53:head# 2024-08-18T21:32:33.618 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:5ee3514b:::existing_30:head# 2024-08-18T21:32:33.618 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:602f83fe:::foo:head# 2024-08-18T21:32:33.618 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:60426cc7:::existing_62:head# 2024-08-18T21:32:33.619 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:628918ca:::existing_79:head# 2024-08-18T21:32:33.619 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-08-18T21:32:33.620 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:67e1780f:::existing_27:head# 2024-08-18T21:32:33.620 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:68143ca1:::existing_71:head# 2024-08-18T21:32:33.621 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:6bac9509:::existing_29:head# 2024-08-18T21:32:33.621 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:6bbec798:::existing_45:head# 2024-08-18T21:32:33.622 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:6cb23a40:::existing_65:head# 2024-08-18T21:32:33.622 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:7055ec5e:::existing_83:head# 2024-08-18T21:32:33.622 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:70a54e2b:::existing_87:head# 2024-08-18T21:32:33.623 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:750ace21:::existing_19:head# 2024-08-18T21:32:33.623 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:78623b1d:::existing_6:head# 2024-08-18T21:32:33.624 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:79881152:::existing_63:head# 2024-08-18T21:32:33.624 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-08-18T21:32:33.624 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:81490677:::existing_76:head# 2024-08-18T21:32:33.625 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-08-18T21:32:33.625 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-08-18T21:32:33.626 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-08-18T21:32:33.626 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:923c5d4d:::existing_5:head# 2024-08-18T21:32:33.627 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:92731e71:::existing_89:head# 2024-08-18T21:32:33.627 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:928205cf:::existing_82:head# 2024-08-18T21:32:33.627 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:93be2267:::existing_24:head# 2024-08-18T21:32:33.628 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:99ceda2c:::existing_44:head# 2024-08-18T21:32:33.628 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:9dd917ee:::existing_26:head# 2024-08-18T21:32:33.629 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:a28805a6:::existing_72:head# 2024-08-18T21:32:33.629 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:a294a8f6:::existing_16:head# 2024-08-18T21:32:33.630 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:a2f5473a:::existing_42:head# 2024-08-18T21:32:33.630 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:a504a533:::existing_90:head# 2024-08-18T21:32:33.630 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:a691337e:::existing_61:head# 2024-08-18T21:32:33.631 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-08-18T21:32:33.632 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:a9fde86e:::existing_18:head# 2024-08-18T21:32:33.632 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:aa309faa:::existing_68:head# 2024-08-18T21:32:33.632 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-08-18T21:32:33.633 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:b279b1fe:::existing_10:head# 2024-08-18T21:32:33.633 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:b343e277:::existing_88:head# 2024-08-18T21:32:33.634 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:b7130e90:::existing_80:head# 2024-08-18T21:32:33.635 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:bc6f7309:::existing_70:head# 2024-08-18T21:32:33.635 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:be8489b7:::existing_38:head# 2024-08-18T21:32:33.635 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-08-18T21:32:33.635 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:bebec877:::existing_74:head# 2024-08-18T21:32:33.636 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:c04347d8:::existing_60:head# 2024-08-18T21:32:33.636 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:c156033e:::existing_100:head# 2024-08-18T21:32:33.637 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-08-18T21:32:33.637 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-08-18T21:32:33.638 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-08-18T21:32:33.638 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-08-18T21:32:33.639 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:ccb3865b:::existing_73:head# 2024-08-18T21:32:33.639 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:cd84e19e:::existing_40:head# 2024-08-18T21:32:33.640 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-08-18T21:32:33.640 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:d41a25c9:::existing_22:head# 2024-08-18T21:32:33.640 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:d7f088da:::existing_91:head# 2024-08-18T21:32:33.641 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-08-18T21:32:33.641 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-08-18T21:32:33.642 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:dd677bd3:::existing_35:head# 2024-08-18T21:32:33.642 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-08-18T21:32:33.643 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:e0e22077:::existing_21:head# 2024-08-18T21:32:33.643 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:e255b710:::existing_55:head# 2024-08-18T21:32:33.644 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:eff1d5db:::existing_14:head# 2024-08-18T21:32:33.644 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:f1b156bf:::existing_23:head# 2024-08-18T21:32:33.644 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:f5ce4199:::existing_34:head# 2024-08-18T21:32:33.645 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:f6809456:::existing_11:head# 2024-08-18T21:32:33.645 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:f699a2a9:::existing_67:head# 2024-08-18T21:32:33.646 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:f9baac43:::existing_25:head# 2024-08-18T21:32:33.646 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:fb76688e:::existing_51:head# 2024-08-18T21:32:33.646 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:fbb92eec:::existing_69:head# 2024-08-18T21:32:33.647 INFO:tasks.workunit.client.0.smithi099.stderr:Export successful 2024-08-18T21:32:33.647 INFO:tasks.workunit.client.0.smithi099.stdout: marking collection for removal 2024-08-18T21:32:33.647 INFO:tasks.workunit.client.0.smithi099.stdout:setting '_remove' omap key 2024-08-18T21:32:33.648 INFO:tasks.workunit.client.0.smithi099.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-08-18T21:32:33.897 INFO:tasks.workunit.client.0.smithi099.stderr:Remove successful 2024-08-18T21:32:34.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.27423.out 2024-08-18T21:32:34.434 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-18T21:32:34.434 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-18T21:32:34.434 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-18T21:32:34.434 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-18T21:32:34.434 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-08-18T21:32:34.434 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.27423.out 2024-08-18T21:32:35.836 INFO:tasks.workunit.client.0.smithi099.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={23=1}} deleted_pools ) 2024-08-18T21:32:35.837 INFO:tasks.workunit.client.0.smithi099.stdout:Importing pgid 1.0 2024-08-18T21:32:35.837 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:023dd3d2:::existing_12:head# 2024-08-18T21:32:35.837 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:35.917 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:04c40a0c:::existing_93:head# 2024-08-18T21:32:35.917 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:35.959 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:064ff5fb:::existing_36:head# 2024-08-18T21:32:35.959 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.000 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:091f9f1d:::existing_94:head# 2024-08-18T21:32:36.001 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.042 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:0a865a93:::existing_54:head# 2024-08-18T21:32:36.043 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.084 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:0fed513c:::existing_41:head# 2024-08-18T21:32:36.084 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.126 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-08-18T21:32:36.126 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.168 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:169157fa:::existing_81:head# 2024-08-18T21:32:36.168 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.210 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:17731382:::existing_58:head# 2024-08-18T21:32:36.210 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.251 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:18a45a39:::existing_43:head# 2024-08-18T21:32:36.251 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.293 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:1c160f01:::existing_66:head# 2024-08-18T21:32:36.293 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.343 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:1dd6c249:::existing_97:head# 2024-08-18T21:32:36.343 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.385 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:227e0c77:::existing_7:head# 2024-08-18T21:32:36.385 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.427 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:23e1b063:::existing_59:head# 2024-08-18T21:32:36.427 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.468 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:28566edb:::existing_78:head# 2024-08-18T21:32:36.469 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.510 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:2a101182:::existing_20:head# 2024-08-18T21:32:36.513 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.552 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:2d1519da:::existing_96:head# 2024-08-18T21:32:36.552 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.594 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:3078b87a:::existing_86:head# 2024-08-18T21:32:36.594 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.636 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:30bde01b:::existing_15:head# 2024-08-18T21:32:36.636 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.677 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:317a7e19:::existing_56:head# 2024-08-18T21:32:36.678 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.719 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-08-18T21:32:36.719 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.761 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:3e16d84b:::existing_48:head# 2024-08-18T21:32:36.761 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.794 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:45fe8203:::existing_75:head# 2024-08-18T21:32:36.795 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.828 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-08-18T21:32:36.828 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.861 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:5298dcdb:::existing_57:head# 2024-08-18T21:32:36.861 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.895 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:52ebaa44:::existing_32:head# 2024-08-18T21:32:36.895 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.928 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:53588912:::existing_9:head# 2024-08-18T21:32:36.928 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.962 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-08-18T21:32:36.962 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:36.995 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:5da0255c:::existing_13:head# 2024-08-18T21:32:36.995 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.037 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:5da2dab8:::existing_53:head# 2024-08-18T21:32:37.037 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.087 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:5ee3514b:::existing_30:head# 2024-08-18T21:32:37.088 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.171 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:602f83fe:::foo:head# 2024-08-18T21:32:37.171 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.213 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:60426cc7:::existing_62:head# 2024-08-18T21:32:37.213 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.255 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:628918ca:::existing_79:head# 2024-08-18T21:32:37.255 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.296 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-08-18T21:32:37.297 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.338 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:67e1780f:::existing_27:head# 2024-08-18T21:32:37.338 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.380 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:68143ca1:::existing_71:head# 2024-08-18T21:32:37.380 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:6bac9509:::existing_29:head# 2024-08-18T21:32:37.432 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.514 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:6bbec798:::existing_45:head# 2024-08-18T21:32:37.514 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.581 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:6cb23a40:::existing_65:head# 2024-08-18T21:32:37.581 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.623 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:7055ec5e:::existing_83:head# 2024-08-18T21:32:37.623 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.665 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:70a54e2b:::existing_87:head# 2024-08-18T21:32:37.665 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.707 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:750ace21:::existing_19:head# 2024-08-18T21:32:37.707 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.748 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:78623b1d:::existing_6:head# 2024-08-18T21:32:37.748 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.790 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:79881152:::existing_63:head# 2024-08-18T21:32:37.790 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.832 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-08-18T21:32:37.832 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.873 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:81490677:::existing_76:head# 2024-08-18T21:32:37.873 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.915 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-08-18T21:32:37.915 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.957 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-08-18T21:32:37.957 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:37.999 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-08-18T21:32:37.999 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.041 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:923c5d4d:::existing_5:head# 2024-08-18T21:32:38.041 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.082 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:92731e71:::existing_89:head# 2024-08-18T21:32:38.083 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.124 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:928205cf:::existing_82:head# 2024-08-18T21:32:38.124 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.166 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:93be2267:::existing_24:head# 2024-08-18T21:32:38.166 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.208 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:99ceda2c:::existing_44:head# 2024-08-18T21:32:38.208 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.250 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:9dd917ee:::existing_26:head# 2024-08-18T21:32:38.250 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.291 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:a28805a6:::existing_72:head# 2024-08-18T21:32:38.292 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.333 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:a294a8f6:::existing_16:head# 2024-08-18T21:32:38.333 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.375 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:a2f5473a:::existing_42:head# 2024-08-18T21:32:38.375 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.417 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:a504a533:::existing_90:head# 2024-08-18T21:32:38.417 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.459 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:a691337e:::existing_61:head# 2024-08-18T21:32:38.459 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.500 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-08-18T21:32:38.501 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.542 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:a9fde86e:::existing_18:head# 2024-08-18T21:32:38.542 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.584 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:aa309faa:::existing_68:head# 2024-08-18T21:32:38.584 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.626 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-08-18T21:32:38.626 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.709 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:b279b1fe:::existing_10:head# 2024-08-18T21:32:38.709 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.751 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:b343e277:::existing_88:head# 2024-08-18T21:32:38.751 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.793 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:b7130e90:::existing_80:head# 2024-08-18T21:32:38.793 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.835 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:bc6f7309:::existing_70:head# 2024-08-18T21:32:38.835 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.876 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:be8489b7:::existing_38:head# 2024-08-18T21:32:38.877 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.918 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-08-18T21:32:38.918 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:38.985 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:bebec877:::existing_74:head# 2024-08-18T21:32:38.985 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.027 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:c04347d8:::existing_60:head# 2024-08-18T21:32:39.027 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.081 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:c156033e:::existing_100:head# 2024-08-18T21:32:39.081 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.177 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-08-18T21:32:39.177 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.219 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-08-18T21:32:39.219 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.261 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-08-18T21:32:39.261 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.302 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-08-18T21:32:39.303 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.344 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:ccb3865b:::existing_73:head# 2024-08-18T21:32:39.344 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.386 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:cd84e19e:::existing_40:head# 2024-08-18T21:32:39.386 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.439 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-08-18T21:32:39.440 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.555 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:d41a25c9:::existing_22:head# 2024-08-18T21:32:39.555 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.629 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:d7f088da:::existing_91:head# 2024-08-18T21:32:39.629 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.663 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-08-18T21:32:39.663 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.696 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-08-18T21:32:39.696 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.729 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:dd677bd3:::existing_35:head# 2024-08-18T21:32:39.729 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.763 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-08-18T21:32:39.763 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.796 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:e0e22077:::existing_21:head# 2024-08-18T21:32:39.796 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.830 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:e255b710:::existing_55:head# 2024-08-18T21:32:39.830 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.863 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:eff1d5db:::existing_14:head# 2024-08-18T21:32:39.863 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.897 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:f1b156bf:::existing_23:head# 2024-08-18T21:32:39.897 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.930 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:f5ce4199:::existing_34:head# 2024-08-18T21:32:39.930 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:39.972 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:f6809456:::existing_11:head# 2024-08-18T21:32:39.972 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:40.014 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:f699a2a9:::existing_67:head# 2024-08-18T21:32:40.014 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:40.055 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:f9baac43:::existing_25:head# 2024-08-18T21:32:40.056 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:40.097 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:fb76688e:::existing_51:head# 2024-08-18T21:32:40.098 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:40.139 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:fbb92eec:::existing_69:head# 2024-08-18T21:32:40.139 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:32:40.181 INFO:tasks.workunit.client.0.smithi099.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-08-18T21:32:40.223 INFO:tasks.workunit.client.0.smithi099.stdout:Import successful 2024-08-18T21:32:40.759 INFO:tasks.workunit.client.0.smithi099.stdout:reviving divergent 1 2024-08-18T21:32:40.759 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-08-18T21:32:40.759 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:32:41.171 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:32:41.172 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-08-18T21:32:29.487479+0000 47'102 47:135 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 no scrub is scheduled 0 0 2024-08-18T21:32:41.172 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:32:41.172 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:32:41.172 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:32:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-08-18T21:32:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:32:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:32:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-18T21:32:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:32:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-18T21:32:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:32:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:32:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:32:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:32:41.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-18T21:32:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-18T21:32:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:32:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:32:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:32:41.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:32:41.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:32:41.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:32:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:32:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:32:41.190 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:32:41.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:32:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:32:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:32:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:32:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:32:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:32:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:32:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:32:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:32:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:32:41.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:32:41.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-18T21:32:41.193 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:32:41.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-18T21:32:41.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:32:41.194 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-18T21:32:41.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-18T21:32:41.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:32:41.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:32:41.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:32:41.240 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:32:41.245+0000 7fa147a017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:32:41.242 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:32:41.245+0000 7fa147a017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:32:41.244 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:32:41.249+0000 7fa147a017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:32:41.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-18T21:32:41.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:32:41.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:32:41.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:32:41.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:32:41.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:32:41.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:32:41.663 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:32:41.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:32:41.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:32:42.098 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2968621215,v1:127.0.0.1:6819/2968621215] [v2:127.0.0.1:6820/2968621215,v1:127.0.0.1:6821/2968621215] exists,up 44b90f3c-ec32-432f-9c24-f78d2330efa6 2024-08-18T21:32:42.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:32:42.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:32:42.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:32:42.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-08-18T21:32:42.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:32:42.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:32:42.100 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:32:42.101 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:32:42.101 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:32:42.101 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:32:42.101 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:32:42.101 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:32:42.101 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:32:42.318 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:32:42.321+0000 7fa147a017c0 -1 Falling back to public interface 2024-08-18T21:32:42.541 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2968621215,v1:127.0.0.1:6819/2968621215] [v2:127.0.0.1:6820/2968621215,v1:127.0.0.1:6821/2968621215] exists,up 44b90f3c-ec32-432f-9c24-f78d2330efa6 2024-08-18T21:32:42.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:32:42.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:32:42.542 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:32:42.542 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-08-18T21:32:45.317 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:32:45.321+0000 7fa147a017c0 -1 osd.1 47 log_to_monitors true 2024-08-18T21:33:02.544 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-08-18T21:33:02.545 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-18T21:33:02.545 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-18T21:33:02.545 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:33:02.545 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:33:02.545 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:33:02.545 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.1.asok 2024-08-18T21:33:02.545 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-08-18T21:33:02.545 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.1.asok dump_ops_in_flight 2024-08-18T21:33:02.660 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:33:02.661 INFO:tasks.workunit.client.0.smithi099.stdout: "ops": [], 2024-08-18T21:33:02.661 INFO:tasks.workunit.client.0.smithi099.stdout: "num_ops": 0 2024-08-18T21:33:02.661 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:33:02.673 INFO:tasks.workunit.client.0.smithi099.stdout:allowing recovery 2024-08-18T21:33:02.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-08-18T21:33:02.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:33:03.091 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:33:03.091 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-08-18T21:32:47.890263+0000 47'102 52:158 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T05:29:07.666099+0000 0 0 2024-08-18T21:33:03.091 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:33:03.091 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:33:03.092 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:33:03.107 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-08-18T21:33:03.107 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-08-18T21:33:03.263 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:33:03.277 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-08-18T21:33:03.277 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-08-18T21:33:03.456 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:33:03.470 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-08-18T21:33:03.470 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-08-18T21:33:03.615 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:33:03.629 INFO:tasks.workunit.client.0.smithi099.stdout:reading divergent objects 2024-08-18T21:33:03.629 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-08-18T21:33:03.630 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-08-18T21:33:04.033 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:33:04.033 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-08-18T21:32:47.890263+0000 47'102 52:158 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:29:59.536320+0000 0'0 2024-08-18T21:29:59.536320+0000 0 0 periodic scrub scheduled @ 2024-08-20T05:29:07.666099+0000 0 0 2024-08-18T21:33:04.033 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:33:04.033 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:33:04.034 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:33:04.050 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-08-18T21:33:04.052 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-08-18T21:33:04.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:33:04.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-08-18T21:33:04.088 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:33:04.088 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-08-18T21:33:04.121 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:33:04.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-08-18T21:33:04.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:33:04.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-08-18T21:33:04.189 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-08-18T21:33:04.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-18T21:33:04.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-08-18T21:33:04.221 INFO:tasks.workunit.client.0.smithi099.stderr:error getting test/newobject_1: (2) No such file or directory 2024-08-18T21:33:04.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-18T21:33:04.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-08-18T21:33:04.254 INFO:tasks.workunit.client.0.smithi099.stderr:error getting test/newobject_2: (2) No such file or directory 2024-08-18T21:33:04.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-08-18T21:33:04.257 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-08-18T21:33:04.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-18T21:33:04.262 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4643.0:1 2024-08-18T21:30:37.562071+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:33:04.262 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: keeping 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4643.0:1 2024-08-18T21:30:37.562071+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:33:04.262 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:33:04.262 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-08-18T21:33:04.262 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: more recent entry found: 41'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4762.0:1 2024-08-18T21:31:42.029676+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-08-18T21:33:04.262 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4632.0:1 2024-08-18T21:30:37.542513+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:33:04.262 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4632.0:1 2024-08-18T21:30:37.542513+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:33:04.262 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:33:04.262 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4642.0:1 2024-08-18T21:30:37.563178+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4642.0:1 2024-08-18T21:30:37.563178+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4642.0:1 2024-08-18T21:30:37.563178+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-08-18T21:33:04.263 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4634.0:1 2024-08-18T21:30:37.547261+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4634.0:1 2024-08-18T21:30:37.547261+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4634.0:1 2024-08-18T21:30:37.547261+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4645.0:1 2024-08-18T21:30:37.566166+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:33:04.264 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: keeping 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4645.0:1 2024-08-18T21:30:37.566166+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:33:04.265 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:33:04.265 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-08-18T21:33:04.265 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-08-18T21:33:04.265 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-08-18T21:33:04.265 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-08-18T21:33:04.265 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'106 (29'1) modify 1:d858c035:::existing_1:head by client.4645.0:1 2024-08-18T21:30:37.566166+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:33:04.265 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-08-18T21:33:04.265 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-08-18T21:33:04.265 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4636.0:1 2024-08-18T21:30:37.550615+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:33:04.266 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4636.0:1 2024-08-18T21:30:37.550615+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:33:04.266 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:33:04.266 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-08-18T21:33:04.266 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-08-18T21:33:04.266 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:32:29.285+0000 7fc3be65a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-08-18T21:33:04.266 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-08-18T21:33:04.266 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-18T21:33:04.269 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-08-18T21:33:04.270 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-18T21:33:04.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-08-18T21:33:04.274 INFO:tasks.workunit.client.0.smithi099.stdout:success 2024-08-18T21:33:04.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.27423.out 2024-08-18T21:33:04.275 INFO:tasks.workunit.client.0.smithi099.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27423.out': No such file or directory 2024-08-18T21:33:04.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-08-18T21:33:04.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-08-18T21:33:04.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-08-18T21:33:04.793 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'test' does not exist 2024-08-18T21:33:04.809 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-08-18T21:33:04.810 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:33:04.810 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:33:04.810 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:33:04.810 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:33:04.811 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:33:05.120 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:33:05.121 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-08-18T21:33:05.121 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-18T21:33:05.121 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:33:05.121 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-18T21:33:05.122 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:33:05.122 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:33:05.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:33:05.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:33:05.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:33:05.128 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:33:05.129 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:33:05.129 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:33:05.129 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:33:05.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:33:05.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:33:05.131 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:33:05.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:33:05.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:33:05.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:33:05.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:33:05.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:33:05.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:33:05.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-18T21:33:05.156 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:33:05.156 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:33:05.156 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:33:05.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27423 2024-08-18T21:33:05.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:33:05.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:33:05.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:33:05.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:33:05.160 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:33:05.160 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:33:05.161 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:33:05.161 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:33:05.161 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:33:05.162 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:33:05.162 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:33:05.163 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:33:05.163 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:33:05.163 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:33:05.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:33:05.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:33:05.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-18T21:33:05.166 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:33:05.166 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:33:05.166 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:33:05.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27423 2024-08-18T21:33:05.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:33:05.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:33:05.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-08-18T21:33:05.168 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-18T21:33:05.168 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:33:05.168 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:33:05.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27423 2024-08-18T21:33:05.170 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-18T21:33:05.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-18T21:33:05.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-18T21:33:05.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-08-18T21:33:05.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-08-18T21:33:05.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-08-18T21:33:05.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-08-18T21:33:05.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-08-18T21:33:05.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-08-18T21:33:05.171 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-08-18T21:33:05.172 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-08-18T21:33:05.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-08-18T21:33:05.172 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:33:05.172 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:33:05.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-08-18T21:33:05.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-08-18T21:33:05.173 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-18T21:33:05.173 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-18T21:33:05.173 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-18T21:33:05.173 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-08-18T21:33:05.173 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-08-18T21:33:05.694 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-18T21:33:05.694 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:33:05.695 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:33:05.695 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:33:05.695 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:33:05.696 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:33:05.696 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:33:05.696 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-18T21:33:05.920 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-18T21:33:05.921 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-18T21:33:05.921 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:33:05.921 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:33:05.921 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-18T21:33:05.922 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-18T21:33:05.923 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:33:05.923 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:33:05.923 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:33:05.923 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:33:05.923 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:33:05.923 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:33:05.923 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-mon.a.asok 2024-08-18T21:33:05.923 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:33:05.924 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27423/ceph-mon.a.asok config get fsid 2024-08-18T21:33:06.034 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-18T21:33:06.035 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:33:06.035 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:33:06.035 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-18T21:33:06.035 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-18T21:33:06.036 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:33:06.036 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:33:06.036 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:33:06.036 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:33:06.036 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:33:06.036 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:33:06.036 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-mon.a.asok 2024-08-18T21:33:06.036 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:33:06.037 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27423/ceph-mon.a.asok config get mon_host 2024-08-18T21:33:06.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-08-18T21:33:06.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-08-18T21:33:06.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-18T21:33:06.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-18T21:33:06.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-18T21:33:06.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-08-18T21:33:06.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-18T21:33:07.078 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-18T21:33:07.078 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:33:07.078 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:33:07.078 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:33:07.078 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:33:07.078 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:33:07.078 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:33:07.079 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:33:07.080 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:33:07.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-08-18T21:33:07.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-08-18T21:33:07.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:33:07.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:33:07.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-18T21:33:07.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:33:07.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-08-18T21:33:07.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:33:07.165 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:33:07.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:33:07.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:33:07.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-08-18T21:33:07.167 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:33:07.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6e2c559a-40e6-4a8a-8afe-cf8a54df7cbc 2024-08-18T21:33:07.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6e2c559a-40e6-4a8a-8afe-cf8a54df7cbc' 2024-08-18T21:33:07.168 INFO:tasks.workunit.client.0.smithi099.stdout:add osd0 6e2c559a-40e6-4a8a-8afe-cf8a54df7cbc 2024-08-18T21:33:07.168 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:33:07.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTaMJmKc2jCxAACrccA7L7ILnLneGbTuf44Q== 2024-08-18T21:33:07.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTaMJmKc2jCxAACrccA7L7ILnLneGbTuf44Q=="}' 2024-08-18T21:33:07.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6e2c559a-40e6-4a8a-8afe-cf8a54df7cbc -i td/divergent-priors/0/new.json 2024-08-18T21:33:07.525 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:33:07.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-08-18T21:33:07.542 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTaMJmKc2jCxAACrccA7L7ILnLneGbTuf44Q== --osd-uuid 6e2c559a-40e6-4a8a-8afe-cf8a54df7cbc 2024-08-18T21:33:07.591 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:07.593+0000 7ff7476b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:07.602 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:07.605+0000 7ff7476b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:07.603 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:07.605+0000 7ff7476b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:07.603 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:07.609+0000 7ff7476b97c0 -1 bdev(0x555ecc0a9c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-08-18T21:33:07.604 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:07.609+0000 7ff7476b97c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-08-18T21:33:13.852 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-08-18T21:33:13.852 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:33:13.854 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd0 key to auth repository 2024-08-18T21:33:13.855 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-18T21:33:13.855 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:33:14.449 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:33:14.450 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-18T21:33:14.450 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:33:14.450 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:33:14.450 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:33:14.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:33:14.491 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:14.497+0000 7f4485ff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:14.514 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:14.517+0000 7f4485ff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:14.515 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:14.521+0000 7f4485ff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:14.945 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-18T21:33:14.945 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:33:14.945 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:33:14.945 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:33:14.946 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:33:14.946 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:33:14.946 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:14.946 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:33:14.946 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:14.946 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:15.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:16.006 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:16.009+0000 7f4485ff17c0 -1 Falling back to public interface 2024-08-18T21:33:16.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:16.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:16.375 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:33:16.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:33:16.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:16.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:16.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:17.816 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:17.816 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:17.816 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:33:17.816 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:33:17.817 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:17.817 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:18.266 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:18.956 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:18.961+0000 7f4485ff17c0 -1 osd.0 0 log_to_monitors true 2024-08-18T21:33:19.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:19.268 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:19.268 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:33:19.269 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:33:19.269 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:19.269 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:19.722 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:20.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:20.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:20.724 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:33:20.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:33:20.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:20.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:21.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:22.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:22.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:22.180 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:33:22.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:33:22.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:22.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:22.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:23.759 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:23.759 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:23.759 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:33:23.759 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:33:23.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:23.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:24.214 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:25.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:25.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:25.215 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:33:25.216 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:33:25.216 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:25.216 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:25.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:26.799 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:26.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:26.800 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:33:26.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:33:26.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:26.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:27.269 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:28.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:28.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:28.272 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:33:28.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:33:28.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:28.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:28.784 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:29.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:29.786 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:29.786 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:33:29.786 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:33:29.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:29.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:30.227 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:31.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:31.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:31.229 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:33:31.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:33:31.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:31.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:31.749 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:32.751 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:32.751 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:32.752 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:33:32.752 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:33:32.752 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:32.752 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:33.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:34.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:34.944 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:34.944 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:33:34.944 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:33:34.944 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:34.945 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:35.699 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:36.701 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:36.702 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:36.702 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:33:36.702 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:33:36.702 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:36.703 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:37.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:38.543 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:38.543 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:38.543 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:33:38.544 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:33:38.544 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:38.544 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:39.777 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:40.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:40.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:40.780 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:33:40.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:33:40.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:40.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:41.587 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:42.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:42.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:42.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:33:42.589 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:33:42.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:42.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:33:43.134 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2216089247,v1:127.0.0.1:6803/2216089247] [v2:127.0.0.1:6804/2216089247,v1:127.0.0.1:6805/2216089247] exists,up 6e2c559a-40e6-4a8a-8afe-cf8a54df7cbc 2024-08-18T21:33:43.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:33:43.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:33:43.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:33:43.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-08-18T21:33:43.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:33:43.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:33:43.138 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:33:43.138 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:33:43.138 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:33:43.138 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:33:43.138 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:33:43.138 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:33:43.138 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:33:43.139 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:33:43.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-08-18T21:33:43.140 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:33:43.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d2abeb81-0f38-419c-ba93-2eea17b6fc91 2024-08-18T21:33:43.141 INFO:tasks.workunit.client.0.smithi099.stdout:add osd1 d2abeb81-0f38-419c-ba93-2eea17b6fc91 2024-08-18T21:33:43.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d2abeb81-0f38-419c-ba93-2eea17b6fc91' 2024-08-18T21:33:43.141 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:33:43.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3aMJms7NbChAAOcht1znp86VM5aQ4h96OWA== 2024-08-18T21:33:43.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3aMJms7NbChAAOcht1znp86VM5aQ4h96OWA=="}' 2024-08-18T21:33:43.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d2abeb81-0f38-419c-ba93-2eea17b6fc91 -i td/divergent-priors/1/new.json 2024-08-18T21:33:43.690 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:33:43.706 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-08-18T21:33:43.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3aMJms7NbChAAOcht1znp86VM5aQ4h96OWA== --osd-uuid d2abeb81-0f38-419c-ba93-2eea17b6fc91 2024-08-18T21:33:43.741 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:43.744+0000 7f9150aad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:43.743 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:43.748+0000 7f9150aad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:43.744 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:43.748+0000 7f9150aad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:43.745 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:43.748+0000 7f9150aad7c0 -1 bdev(0x560132ad5c00 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-08-18T21:33:43.745 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:43.748+0000 7f9150aad7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-08-18T21:33:50.675 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-08-18T21:33:50.675 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:33:50.676 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd1 key to auth repository 2024-08-18T21:33:50.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-18T21:33:50.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:33:51.261 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:33:51.262 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-18T21:33:51.262 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:33:51.262 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:33:51.262 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:33:51.263 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:33:51.297 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:51.300+0000 7f5e5cb347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:51.324 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:51.328+0000 7f5e5cb347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:51.326 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:51.328+0000 7f5e5cb347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:33:51.763 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-18T21:33:51.764 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:33:51.764 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:33:51.764 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:33:51.764 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:33:51.764 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:51.764 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:33:51.764 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:33:51.765 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:51.765 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:33:52.279 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:52.342 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:52.344+0000 7f5e5cb347c0 -1 Falling back to public interface 2024-08-18T21:33:53.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:53.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:53.281 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:33:53.282 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:33:53.282 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:53.282 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:33:53.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:54.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:54.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:54.728 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:33:54.728 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:33:54.728 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:54.728 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:33:55.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:55.236 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:33:55.240+0000 7f5e5cb347c0 -1 osd.1 0 log_to_monitors true 2024-08-18T21:33:56.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:56.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:56.157 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:33:56.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:33:56.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:56.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:33:56.635 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:57.637 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:57.637 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:57.637 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:33:57.638 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:33:57.638 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:57.638 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:33:58.315 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:33:59.317 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:33:59.317 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:33:59.317 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:33:59.318 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:33:59.318 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:33:59.318 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:33:59.766 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:00.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:00.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:00.768 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:34:00.769 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:34:00.769 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:00.769 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:01.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:02.276 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:02.276 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:02.276 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:34:02.277 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:34:02.277 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:02.277 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:02.871 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:03.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:03.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:03.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:34:03.874 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:34:03.874 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:03.874 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:04.414 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:05.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:05.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:05.416 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:34:05.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:34:05.417 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:05.417 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:05.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:06.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:06.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:06.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:34:06.895 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:34:06.896 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:06.896 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:07.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:08.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:08.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:08.365 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:34:08.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:34:08.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:08.366 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:08.901 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:09.901 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:09.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:09.902 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:34:09.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:34:09.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:09.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:10.994 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:11.996 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:11.996 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:11.996 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:34:11.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:34:11.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:11.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:13.352 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:14.354 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:14.354 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:14.354 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:34:14.355 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:34:14.355 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:14.355 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:15.332 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:16.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:16.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:16.334 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:34:16.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:34:16.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:16.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:17.250 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:18.251 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:18.251 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:18.252 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:34:18.252 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:34:18.252 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:18.252 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:19.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:20.062 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:20.062 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:20.063 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:34:20.063 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:34:20.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:20.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:20.692 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:21.693 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:21.693 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:21.693 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:34:21.694 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:34:21.694 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:21.694 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:34:22.164 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2289922869,v1:127.0.0.1:6811/2289922869] [v2:127.0.0.1:6812/2289922869,v1:127.0.0.1:6813/2289922869] exists,up d2abeb81-0f38-419c-ba93-2eea17b6fc91 2024-08-18T21:34:22.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:34:22.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:34:22.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:34:22.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:34:22.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:34:22.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:34:22.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:34:22.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:34:22.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:34:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:34:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:34:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:34:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:34:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:34:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:34:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-08-18T21:34:22.172 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:34:22.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=33c0de0c-1a9c-4bb2-9e7b-3a49e559e1e4 2024-08-18T21:34:22.172 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 33c0de0c-1a9c-4bb2-9e7b-3a49e559e1e4' 2024-08-18T21:34:22.172 INFO:tasks.workunit.client.0.smithi099.stdout:add osd2 33c0de0c-1a9c-4bb2-9e7b-3a49e559e1e4 2024-08-18T21:34:22.172 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:34:22.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDeaMJmIcG6CxAAPh8tMM92Rh42rM/0368gYQ== 2024-08-18T21:34:22.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDeaMJmIcG6CxAAPh8tMM92Rh42rM/0368gYQ=="}' 2024-08-18T21:34:22.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 33c0de0c-1a9c-4bb2-9e7b-3a49e559e1e4 -i td/divergent-priors/2/new.json 2024-08-18T21:34:22.694 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:34:22.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-08-18T21:34:22.712 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDeaMJmIcG6CxAAPh8tMM92Rh42rM/0368gYQ== --osd-uuid 33c0de0c-1a9c-4bb2-9e7b-3a49e559e1e4 2024-08-18T21:34:22.748 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:34:22.751+0000 7f9e957327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:34:22.775 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:34:22.775+0000 7f9e957327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:34:22.776 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:34:22.779+0000 7f9e957327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:34:22.777 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:34:22.779+0000 7f9e957327c0 -1 bdev(0x55cf33e6fc00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-08-18T21:34:22.777 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:34:22.779+0000 7f9e957327c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-08-18T21:34:29.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-08-18T21:34:29.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:34:29.350 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd2 key to auth repository 2024-08-18T21:34:29.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-18T21:34:29.351 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:34:29.942 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:34:29.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-18T21:34:29.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:34:29.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:34:29.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:34:29.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:34:29.985 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:34:29.987+0000 7f0c833f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:34:30.011 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:34:30.011+0000 7f0c833f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:34:30.012 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:34:30.015+0000 7f0c833f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:34:30.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-18T21:34:30.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:34:30.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:34:30.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:34:30.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:34:30.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:30.381 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:34:30.382 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:34:30.382 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:30.382 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:30.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:31.506 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:34:31.507+0000 7f0c833f77c0 -1 Falling back to public interface 2024-08-18T21:34:31.827 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:31.827 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:31.827 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:34:31.827 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:34:31.828 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:31.828 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:32.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:33.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:33.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:33.312 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:34:33.312 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:34:33.312 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:33.312 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:33.758 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:34.558 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:34:34.559+0000 7f0c833f77c0 -1 osd.2 0 log_to_monitors true 2024-08-18T21:34:34.759 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:34.759 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:34.759 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:34:34.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:34:34.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:34.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:35.413 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:36.414 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:36.414 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:36.415 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:34:36.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:34:36.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:36.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:36.881 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:37.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:37.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:37.883 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:34:37.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:34:37.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:37.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:38.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:39.492 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:39.492 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:39.492 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:34:39.493 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:34:39.493 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:39.493 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:39.957 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:40.959 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:40.959 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:40.959 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:34:40.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:34:40.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:40.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:41.500 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:42.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:42.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:42.503 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:34:42.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:34:42.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:42.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:42.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:43.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:43.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:43.973 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:34:43.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:34:43.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:43.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:44.496 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:45.498 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:45.498 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:45.499 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:34:45.499 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:34:45.499 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:45.499 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:46.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:47.001 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:47.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:47.002 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:34:47.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:34:47.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:47.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:47.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:48.548 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:48.549 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:48.549 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:34:48.549 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:34:48.549 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:48.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:49.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:50.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:50.136 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:34:50.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:50.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:34:50.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:50.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:51.210 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:52.212 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:52.212 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:52.212 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:34:52.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:34:52.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:52.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:53.090 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:54.092 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:54.092 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:54.092 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:34:54.093 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:34:54.093 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:54.093 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:55.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:56.068 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:56.068 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:56.068 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:34:56.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:34:56.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:56.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:56.993 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:57.995 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:57.995 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:57.995 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:34:57.996 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:34:57.996 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:57.996 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:34:58.783 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:34:59.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:34:59.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:34:59.785 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:34:59.786 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:34:59.786 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:34:59.786 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:35:00.221 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2191126558,v1:127.0.0.1:6819/2191126558] [v2:127.0.0.1:6820/2191126558,v1:127.0.0.1:6821/2191126558] exists,up 33c0de0c-1a9c-4bb2-9e7b-3a49e559e1e4 2024-08-18T21:35:00.222 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:35:00.222 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:35:00.222 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:35:00.222 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-08-18T21:35:01.006 INFO:tasks.workunit.client.0.smithi099.stderr:noout is set 2024-08-18T21:35:01.023 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-08-18T21:35:01.852 INFO:tasks.workunit.client.0.smithi099.stderr:noin is set 2024-08-18T21:35:01.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-08-18T21:35:02.857 INFO:tasks.workunit.client.0.smithi099.stderr:nodown is set 2024-08-18T21:35:02.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-08-18T21:35:02.877 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-08-18T21:35:03.593 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'test' already exists 2024-08-18T21:35:03.610 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-18T21:35:04.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-08-18T21:35:05.501 INFO:tasks.workunit.client.0.smithi099.stderr:set pool 1 size to 3 2024-08-18T21:35:05.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-08-18T21:35:06.338 INFO:tasks.workunit.client.0.smithi099.stderr:set pool 1 min_size to 2 2024-08-18T21:35:06.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-08-18T21:35:07.271 INFO:tasks.workunit.client.0.smithi099.stderr:set pool 1 pg_autoscale_mode to on 2024-08-18T21:35:07.293 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-08-18T21:35:07.294 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-08-18T21:35:07.296 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1724016907 2024-08-18T21:35:07.296 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-08-18T21:35:07.296 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-08-18T21:35:07.296 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-08-18T21:35:07.296 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:35:07.297 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:35:07.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:35:07.894 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:35:07.894 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:35:07.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:35:07.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:35:07.894 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:35:08.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836505 2024-08-18T21:35:08.072 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836505 2024-08-18T21:35:08.072 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836505' 2024-08-18T21:35:08.072 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:35:08.072 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:35:08.240 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672983 2024-08-18T21:35:08.240 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672983 2024-08-18T21:35:08.240 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836505 1-42949672983' 2024-08-18T21:35:08.240 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:35:08.241 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:35:08.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509445 2024-08-18T21:35:08.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509445 2024-08-18T21:35:08.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509445' 2024-08-18T21:35:08.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:35:08.409 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836505 2024-08-18T21:35:08.409 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:35:08.410 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:35:08.410 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836505 2024-08-18T21:35:08.411 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:35:08.412 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836505 2024-08-18T21:35:08.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836505 2024-08-18T21:35:08.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-08-18T21:35:08.412 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:35:08.852 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836503 -lt 21474836505 2024-08-18T21:35:08.852 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:35:09.854 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:35:09.854 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:35:10.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836506 -lt 21474836505 2024-08-18T21:35:10.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:35:10.303 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-08-18T21:35:10.303 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:35:10.304 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:35:10.305 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672983 2024-08-18T21:35:10.305 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:35:10.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672983 2024-08-18T21:35:10.306 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672983 2024-08-18T21:35:10.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-08-18T21:35:10.307 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:35:10.754 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672983 -lt 42949672983 2024-08-18T21:35:10.754 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:35:10.755 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509445 2024-08-18T21:35:10.756 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:35:10.758 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:35:10.758 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509445 2024-08-18T21:35:10.759 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:35:10.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509445 2024-08-18T21:35:10.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-08-18T21:35:10.760 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509445 2024-08-18T21:35:10.761 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:35:11.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509445 -lt 64424509445 2024-08-18T21:35:11.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-08-18T21:35:11.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:35:11.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:35:11.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:35:11.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-18T21:35:11.190 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:35:11.190 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:35:11.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:35:11.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:35:11.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:35:11.288 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-18T21:35:11.288 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:35:11.288 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:35:11.288 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:35:11.288 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:35:11.288 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:35:11.703 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:35:11.704 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:35:11.704 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:35:11.704 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:35:11.704 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:35:11.704 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:35:11.872 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836508 2024-08-18T21:35:11.872 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836508 2024-08-18T21:35:11.872 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836508' 2024-08-18T21:35:11.872 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:35:11.872 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:35:12.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672986 2024-08-18T21:35:12.036 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672986 2024-08-18T21:35:12.036 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836508 1-42949672986' 2024-08-18T21:35:12.036 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:35:12.036 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:35:12.225 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509448 2024-08-18T21:35:12.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509448 2024-08-18T21:35:12.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836508 1-42949672986 2-64424509448' 2024-08-18T21:35:12.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:35:12.226 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-08-18T21:35:12.226 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:35:12.227 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:35:12.227 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836508 2024-08-18T21:35:12.228 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:35:12.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836508 2024-08-18T21:35:12.229 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836508 2024-08-18T21:35:12.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-08-18T21:35:12.229 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:35:12.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836508 -lt 21474836508 2024-08-18T21:35:12.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:35:12.661 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672986 2024-08-18T21:35:12.661 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:35:12.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:35:12.663 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672986 2024-08-18T21:35:12.664 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:35:12.665 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672986 2024-08-18T21:35:12.666 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672986 2024-08-18T21:35:12.666 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-08-18T21:35:12.666 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:35:13.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672984 -lt 42949672986 2024-08-18T21:35:13.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:35:14.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:35:14.098 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:35:14.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672986 -lt 42949672986 2024-08-18T21:35:14.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:35:14.679 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509448 2024-08-18T21:35:14.679 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:35:14.680 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:35:14.681 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509448 2024-08-18T21:35:14.681 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:35:14.682 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509448 2024-08-18T21:35:14.682 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-08-18T21:35:14.683 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509448 2024-08-18T21:35:14.683 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:35:15.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509448 -lt 64424509448 2024-08-18T21:35:15.305 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:35:15.306 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:35:15.306 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:35:15.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:35:15.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:35:15.885 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:35:15.886 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:35:15.886 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:35:15.886 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:35:15.886 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:35:15.886 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:35:16.289 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-18T21:35:16.289 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:35:16.290 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:35:16.290 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:35:16.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 32 2024-08-18T21:35:16.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' -1 2024-08-18T21:35:16.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-18T21:35:16.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-08-18T21:35:16.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:35:16.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-18T21:35:16.995 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:35:16.996 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:35:16.996 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:35:16.996 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:35:16.996 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:35:16.996 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:35:16.997 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:35:16.997 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:35:17.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-18T21:35:17.449 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:35:17.449 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:35:17.449 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:35:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 32 2024-08-18T21:35:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-08-18T21:35:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:35:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:35:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:35:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:35:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:35:18.025 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:35:18.025 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:35:18.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:35:18.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:35:18.640 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-18T21:35:18.640 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:35:18.640 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-18T21:35:18.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:35:18.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:35:18.842 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:35:18.842 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:35:18.842 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:35:18.842 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:35:18.843 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:35:18.843 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:35:19.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=24 2024-08-18T21:35:19.306 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:35:19.306 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:35:19.306 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:35:19.939 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 24 = 32 2024-08-18T21:35:19.939 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 24 '!=' 1 2024-08-18T21:35:19.939 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-18T21:35:19.939 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=24 2024-08-18T21:35:19.939 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:35:19.939 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-18T21:35:20.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:35:20.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:35:20.041 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:35:20.041 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:35:20.041 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:35:20.041 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:35:20.042 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:35:20.042 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:35:20.544 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=24 2024-08-18T21:35:20.544 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:35:20.545 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:35:20.545 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:35:21.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 24 = 32 2024-08-18T21:35:21.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 24 '!=' 24 2024-08-18T21:35:21.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:35:21.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:35:21.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:35:21.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:35:21.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:35:21.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:35:21.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:35:21.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:35:21.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:35:21.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-08-18T21:35:21.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:35:21.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-18T21:35:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:35:21.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:35:21.975 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:35:21.975 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:35:21.975 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:35:21.975 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:35:21.976 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:35:21.976 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:35:22.474 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=24 2024-08-18T21:35:22.475 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:35:22.475 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:35:22.475 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:35:23.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 24 = 32 2024-08-18T21:35:23.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 24 '!=' 24 2024-08-18T21:35:23.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:35:23.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:35:23.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:35:23.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:35:23.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:35:23.075 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:35:23.075 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:35:23.717 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:35:23.718 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:35:23.718 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-08-18T21:35:23.718 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:35:23.718 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-18T21:35:24.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:35:24.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:35:24.120 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:35:24.120 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:35:24.121 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:35:24.121 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:35:24.121 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:35:24.121 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:35:24.554 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-08-18T21:35:24.555 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:35:24.555 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:35:24.556 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:35:25.113 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-08-18T21:35:25.113 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:35:25.113 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:35:25.114 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-08-18T21:35:25.114 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-08-18T21:35:25.523 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:35:25.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=0 2024-08-18T21:35:25.540 INFO:tasks.workunit.client.0.smithi099.stdout:primary and soon to be divergent is 0 2024-08-18T21:35:25.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 0' 2024-08-18T21:35:25.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:35:25.953 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:35:26.191 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.831600+0000 0'0 36:40 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.191 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:18.524452+0000 0'0 36:52 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:22:51.247039+0000 0 0 2024-08-18T21:35:26.191 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:21.532043+0000 0'0 36:52 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:28:36.956355+0000 0 0 2024-08-18T21:35:26.191 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:15.144456+0000 0'0 36:44 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.191 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.131394+0000 0'0 36:50 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.191 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.084859+0000 0'0 36:50 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.191 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.243024+0000 0'0 36:50 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.191 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.242986+0000 0'0 36:50 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.191 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:15.142254+0000 0'0 36:44 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:24.402639+0000 0'0 36:48 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:25:26.732185+0000 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:15.137475+0000 0'0 36:44 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.084810+0000 0'0 36:58 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:23:43.334292+0000 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:21.097125+0000 0'0 36:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:54:29.086185+0000 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:15.143430+0000 0'0 36:44 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:20.431242+0000 0'0 36:48 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:22:05.930717+0000 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.832274+0000 0'0 36:40 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:15.449055+0000 0'0 36:52 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-19T23:58:49.929086+0000 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:21.383299+0000 0'0 36:48 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:51:46.629110+0000 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.242995+0000 0'0 36:50 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:23.406346+0000 0'0 36:48 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:37:27.621837+0000 0 0 2024-08-18T21:35:26.192 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.831352+0000 0'0 36:40 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:22.093401+0000 0'0 36:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T05:03:13.803155+0000 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:23.048047+0000 0'0 36:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:57:11.387863+0000 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:15.138409+0000 0'0 36:44 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.830895+0000 0'0 36:40 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:15.143410+0000 0'0 36:44 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.516597+0000 0'0 36:52 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:28:31.243352+0000 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:20.121349+0000 0'0 36:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:09:02.807588+0000 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:22.401149+0000 0'0 36:48 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:37:45.466705+0000 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:20.571486+0000 0'0 36:52 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:58:10.797373+0000 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.241992+0000 0'0 36:50 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:35:19.831756+0000 0'0 36:40 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:03.313989+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 queued for scrub 0 0 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:35:26.193 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:35:26.195 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:35:26.195 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-08-18T21:35:26.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1724016925 2024-08-18T21:35:26.195 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1724016925 - 1724016907 2024-08-18T21:35:26.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=18 2024-08-18T21:35:26.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 18 -gt 300 ']' 2024-08-18T21:35:26.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 0 -le -1 ']' 2024-08-18T21:35:26.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-08-18T21:35:26.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-08-18T21:35:26.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 0 ']' 2024-08-18T21:35:26.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-08-18T21:35:26.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-08-18T21:35:26.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 0 ']' 2024-08-18T21:35:26.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 1' 2024-08-18T21:35:26.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-08-18T21:35:26.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 0 ']' 2024-08-18T21:35:26.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 1 2' 2024-08-18T21:35:26.196 INFO:tasks.workunit.client.0.smithi099.stdout:writing initial objects 2024-08-18T21:35:26.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-08-18T21:35:26.197 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-08-18T21:35:26.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:26.197 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-08-18T21:35:26.270 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:26.270 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-08-18T21:35:26.467 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:26.467 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-08-18T21:35:26.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:26.640 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-08-18T21:35:26.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:26.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-08-18T21:35:27.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:27.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-08-18T21:35:27.327 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:27.327 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-08-18T21:35:27.537 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:27.538 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-08-18T21:35:27.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:27.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-08-18T21:35:27.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:27.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-08-18T21:35:28.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:28.141 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-08-18T21:35:28.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:28.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-08-18T21:35:28.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:28.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-08-18T21:35:28.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:28.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-08-18T21:35:28.994 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:28.994 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-08-18T21:35:29.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:29.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-08-18T21:35:29.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:29.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-08-18T21:35:29.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:29.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-08-18T21:35:29.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:29.758 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-08-18T21:35:29.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:29.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-08-18T21:35:30.084 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:30.084 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-08-18T21:35:30.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:30.245 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-08-18T21:35:30.446 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:30.447 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-08-18T21:35:30.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:30.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-08-18T21:35:30.897 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:30.897 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-08-18T21:35:31.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:31.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-08-18T21:35:31.458 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:31.459 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-08-18T21:35:31.651 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:31.651 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-08-18T21:35:31.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:31.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-08-18T21:35:32.086 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:32.086 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-08-18T21:35:32.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:32.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-08-18T21:35:32.497 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:32.497 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-08-18T21:35:32.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:32.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-08-18T21:35:32.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:32.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-08-18T21:35:33.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:33.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-08-18T21:35:33.335 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:33.335 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-08-18T21:35:33.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:33.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-08-18T21:35:33.771 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:33.771 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-08-18T21:35:33.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:33.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-08-18T21:35:34.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:34.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-08-18T21:35:34.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:34.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-08-18T21:35:34.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:34.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-08-18T21:35:34.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:34.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-08-18T21:35:34.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:34.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-08-18T21:35:35.062 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:35.062 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-08-18T21:35:35.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:35.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-08-18T21:35:35.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:35.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-08-18T21:35:35.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:35.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-08-18T21:35:35.765 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:35.765 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-08-18T21:35:35.950 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:35.950 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-08-18T21:35:36.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:36.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-08-18T21:35:36.301 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:36.301 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-08-18T21:35:36.511 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:36.512 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-08-18T21:35:36.686 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:36.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-08-18T21:35:36.854 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:36.854 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-08-18T21:35:37.021 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:37.022 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-08-18T21:35:37.181 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:37.181 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-08-18T21:35:37.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:37.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-08-18T21:35:37.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:37.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-08-18T21:35:37.692 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:37.692 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-08-18T21:35:37.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:37.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-08-18T21:35:38.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:38.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-08-18T21:35:38.236 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:38.236 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-08-18T21:35:38.394 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:38.395 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-08-18T21:35:38.580 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:38.580 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-08-18T21:35:38.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:38.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-08-18T21:35:38.956 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:38.956 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-08-18T21:35:39.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:39.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-08-18T21:35:39.299 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:39.299 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-08-18T21:35:39.467 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:39.467 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-08-18T21:35:39.635 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:39.635 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-08-18T21:35:39.810 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:39.810 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-08-18T21:35:39.978 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:39.978 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-08-18T21:35:40.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:40.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-08-18T21:35:40.354 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:40.354 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-08-18T21:35:40.532 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:40.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-08-18T21:35:40.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:40.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-08-18T21:35:40.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:40.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-08-18T21:35:41.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:41.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-08-18T21:35:41.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:41.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-08-18T21:35:41.404 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:41.404 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-08-18T21:35:41.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:41.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-08-18T21:35:41.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:41.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-08-18T21:35:41.924 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:41.924 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-08-18T21:35:42.107 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:42.107 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-08-18T21:35:42.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:42.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-08-18T21:35:42.500 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:42.500 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-08-18T21:35:42.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:42.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-08-18T21:35:42.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:42.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-08-18T21:35:43.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:43.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-08-18T21:35:43.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:43.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-08-18T21:35:43.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:43.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-08-18T21:35:43.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:43.548 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-08-18T21:35:43.732 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:43.732 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-08-18T21:35:43.899 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:43.899 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-08-18T21:35:44.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:44.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-08-18T21:35:44.251 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:44.252 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-08-18T21:35:44.445 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:44.445 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-08-18T21:35:44.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:44.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-08-18T21:35:44.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-08-18T21:35:44.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-08-18T21:35:44.963 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-18T21:35:44.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-08-18T21:35:44.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:35:44.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:35:44.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:35:44.964 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-18T21:35:44.964 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:35:44.965 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:35:44.965 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:35:44.965 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:35:44.965 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:35:45.042 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-18T21:35:45.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:35:45.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:35:45.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:35:45.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:35:45.043 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:35:45.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:35:45.540 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:35:45.540 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:35:45.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:35:45.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:35:45.541 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:35:45.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836517 2024-08-18T21:35:45.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836517 2024-08-18T21:35:45.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836517' 2024-08-18T21:35:45.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:35:45.723 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:35:45.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672994 2024-08-18T21:35:45.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672994 2024-08-18T21:35:45.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836517 1-42949672994' 2024-08-18T21:35:45.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:35:45.892 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:35:46.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509457 2024-08-18T21:35:46.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509457 2024-08-18T21:35:46.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836517 1-42949672994 2-64424509457' 2024-08-18T21:35:46.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:35:46.056 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836517 2024-08-18T21:35:46.056 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:35:46.056 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:35:46.057 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836517 2024-08-18T21:35:46.057 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:35:46.058 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836517 2024-08-18T21:35:46.059 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836517 2024-08-18T21:35:46.059 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-08-18T21:35:46.059 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:35:46.498 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836517 -lt 21474836517 2024-08-18T21:35:46.498 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:35:46.499 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672994 2024-08-18T21:35:46.499 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:35:46.500 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:35:46.501 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672994 2024-08-18T21:35:46.502 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:35:46.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672994 2024-08-18T21:35:46.503 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672994 2024-08-18T21:35:46.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-08-18T21:35:46.503 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:35:46.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672994 -lt 42949672994 2024-08-18T21:35:46.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:35:46.941 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:35:46.941 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509457 2024-08-18T21:35:46.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:35:46.943 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509457 2024-08-18T21:35:46.943 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:35:46.944 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509457 2024-08-18T21:35:46.944 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-08-18T21:35:46.944 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509457 2024-08-18T21:35:46.944 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:35:47.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509455 -lt 64424509457 2024-08-18T21:35:47.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:35:48.366 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:35:48.367 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:35:48.792 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509457 -lt 64424509457 2024-08-18T21:35:48.793 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:35:48.793 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:35:48.794 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:35:49.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-08-18T21:35:49.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:35:49.392 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:35:49.392 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:35:49.392 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:35:49.392 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:35:49.392 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:35:49.392 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:35:49.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-08-18T21:35:49.805 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:35:49.805 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:35:49.805 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:35:50.410 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-08-18T21:35:50.410 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:35:50.410 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:35:50.410 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-08-18T21:35:50.411 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-08-18T21:35:50.411 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-08-18T21:35:50.411 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-08-18T21:35:50.411 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-18T21:35:50.900 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-08-18T21:35:50.900 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 1 2' 2024-08-18T21:35:50.900 INFO:tasks.workunit.client.0.smithi099.stdout:blackholing osds 1 2 2024-08-18T21:35:50.900 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:35:51.339 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:35:51.340 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:24.992473+0000 36'3 36:51 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:43:18.899770+0000 0 0 2024-08-18T21:35:51.340 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:18.524452+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:22:51.247039+0000 0 0 2024-08-18T21:35:51.340 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:35:21.532043+0000 36'4 36:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:28:36.956355+0000 0 0 2024-08-18T21:35:51.340 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:25.594716+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T09:21:46.250195+0000 0 0 2024-08-18T21:35:51.340 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:29.535748+0000 36'3 36:61 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:19:51.726053+0000 1 0 2024-08-18T21:35:51.340 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:35:28.529528+0000 36'5 36:63 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T03:41:16.961736+0000 2 0 2024-08-18T21:35:51.340 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:33.539279+0000 36'3 36:61 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-20T09:25:31.831796+0000 0 0 2024-08-18T21:35:51.340 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:35:31.655003+0000 36'2 36:60 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:48:14.349444+0000 1 0 2024-08-18T21:35:51.340 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:35:27.561149+0000 36'4 36:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T01:57:12.845572+0000 0 0 2024-08-18T21:35:51.340 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-08-18T21:35:24.402639+0000 36'6 36:54 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:25:26.732185+0000 0 0 2024-08-18T21:35:51.340 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:35:24.577380+0000 36'2 36:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:49:54.996068+0000 0 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:35:19.084810+0000 36'5 36:63 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:23:43.334292+0000 0 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:35:21.097125+0000 36'1 36:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:54:29.086185+0000 0 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:22.539811+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:38:14.898239+0000 0 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:35:20.431242+0000 36'2 36:50 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:22:05.930717+0000 0 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:35:26.929556+0000 36'2 36:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:16:28.438167+0000 0 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:15.449055+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-19T23:58:49.929086+0000 0 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:21.383299+0000 36'3 36:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:51:46.629110+0000 0 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:35:32.549877+0000 36'4 36:62 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:40:28.398677+0000 2 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:35:23.406346+0000 36'4 36:52 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:37:27.621837+0000 0 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:24.038460+0000 36'3 36:51 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:13:50.898681+0000 0 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:22.093401+0000 36'3 36:51 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T05:03:13.803155+0000 0 0 2024-08-18T21:35:51.341 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:35:23.048047+0000 36'1 36:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:57:11.387863+0000 0 0 2024-08-18T21:35:51.342 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:23.572198+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:43:59.581693+0000 0 0 2024-08-18T21:35:51.342 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:25.419256+0000 36'3 36:51 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:20:44.147140+0000 0 0 2024-08-18T21:35:51.342 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:26.643936+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:16:18.870570+0000 0 0 2024-08-18T21:35:51.342 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:35:19.516597+0000 36'4 36:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:28:31.243352+0000 0 0 2024-08-18T21:35:51.342 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:35:20.121349+0000 36'1 36:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:09:02.807588+0000 0 0 2024-08-18T21:35:51.342 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:22.401149+0000 36'3 36:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:37:45.466705+0000 0 0 2024-08-18T21:35:51.342 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:35:20.571486+0000 36'2 36:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:58:10.797373+0000 0 0 2024-08-18T21:35:51.342 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:35:30.624788+0000 36'4 36:62 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:57:59.352479+0000 1 0 2024-08-18T21:35:51.342 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:35:25.958957+0000 36'5 36:53 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:53:10.798788+0000 0 0 2024-08-18T21:35:51.342 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:35:51.342 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:35:51.343 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:35:51.355 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-08-18T21:35:51.356 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.1 2024-08-18T21:35:51.356 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-18T21:35:51.356 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-18T21:35:51.357 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:35:51.357 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:35:51.357 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:35:51.357 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.1.asok 2024-08-18T21:35:51.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-08-18T21:35:51.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.1.asok config set objectstore_blackhole 1 2024-08-18T21:35:51.482 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:35:51.482 INFO:tasks.workunit.client.0.smithi099.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-18T21:35:51.482 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:35:51.494 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-08-18T21:35:51.495 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-08-18T21:35:51.495 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-18T21:35:51.495 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-18T21:35:51.496 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:35:51.496 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:35:51.496 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:35:51.496 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.2.asok 2024-08-18T21:35:51.497 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-08-18T21:35:51.497 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.2.asok config set objectstore_blackhole 1 2024-08-18T21:35:51.600 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:35:51.600 INFO:tasks.workunit.client.0.smithi099.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-18T21:35:51.600 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:35:51.610 INFO:tasks.workunit.client.0.smithi099.stdout:create new divergent objects 2024-08-18T21:35:51.611 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-08-18T21:35:51.611 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-08-18T21:35:51.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-18T21:35:51.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-18T21:35:51.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-08-18T21:35:51.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-08-18T21:35:51.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-08-18T21:35:51.615 INFO:tasks.workunit.client.0.smithi099.stdout:writing divergent objects 2024-08-18T21:35:51.615 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-08-18T21:35:51.615 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-08-18T21:35:51.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-08-18T21:35:51.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-08-18T21:35:51.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-08-18T21:35:51.616 INFO:tasks.workunit.client.0.smithi099.stdout:remove divergent objects 2024-08-18T21:35:51.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-08-18T21:35:51.617 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-08-18T21:35:51.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-08-18T21:35:51.618 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-08-18T21:35:51.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-08-18T21:35:51.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-08-18T21:35:51.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-08-18T21:35:51.621 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-08-18T21:35:51.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-08-18T21:35:51.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-08-18T21:35:51.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-08-18T21:36:01.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-08-18T21:36:01.630 INFO:tasks.workunit.client.0.smithi099.stderr:rados: no process found 2024-08-18T21:36:01.630 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-08-18T21:36:01.630 INFO:tasks.workunit.client.0.smithi099.stdout:killing all the osds 2024-08-18T21:36:01.630 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:36:02.058 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:36:02.058 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:24.992473+0000 36'3 36:51 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:43:18.899770+0000 0 0 2024-08-18T21:36:02.058 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:18.524452+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:22:51.247039+0000 0 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:35:21.532043+0000 36'4 36:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:28:36.956355+0000 0 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:25.594716+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T09:21:46.250195+0000 0 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:29.535748+0000 36'3 36:61 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:19:51.726053+0000 1 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 0 0 0 3435 0 0 6 0 6 active+clean 2024-08-18T21:35:28.529528+0000 36'6 36:64 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T03:41:16.961736+0000 2 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:33.539279+0000 36'3 36:61 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-20T09:25:31.831796+0000 0 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:35:31.655003+0000 36'2 36:60 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:48:14.349444+0000 1 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:35:27.561149+0000 36'4 36:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T01:57:12.845572+0000 0 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-08-18T21:35:24.402639+0000 36'6 36:54 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:25:26.732185+0000 0 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:35:24.577380+0000 36'2 36:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:49:54.996068+0000 0 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:35:19.084810+0000 36'5 36:63 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:23:43.334292+0000 0 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:35:21.097125+0000 36'1 36:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:54:29.086185+0000 0 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:22.539811+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:38:14.898239+0000 0 0 2024-08-18T21:36:02.059 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:35:20.431242+0000 36'2 36:50 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:22:05.930717+0000 0 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:35:26.929556+0000 36'2 36:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:16:28.438167+0000 0 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:15.449055+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-19T23:58:49.929086+0000 0 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:21.383299+0000 36'3 36:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:51:46.629110+0000 0 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 0 0 0 1851 0 0 6 0 6 active+clean 2024-08-18T21:35:32.549877+0000 36'6 36:64 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:40:28.398677+0000 2 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:35:23.406346+0000 36'4 36:52 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:37:27.621837+0000 0 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:24.038460+0000 36'3 36:51 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:13:50.898681+0000 0 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 2 0 0 0 0 1234 0 0 4 0 4 active+clean 2024-08-18T21:35:22.093401+0000 36'4 36:52 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T05:03:13.803155+0000 0 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:35:23.048047+0000 36'1 36:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:57:11.387863+0000 0 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:23.572198+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:43:59.581693+0000 0 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:25.419256+0000 36'3 36:51 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:20:44.147140+0000 0 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:26.643936+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:16:18.870570+0000 0 0 2024-08-18T21:36:02.060 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:35:19.516597+0000 36'4 36:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:28:31.243352+0000 0 0 2024-08-18T21:36:02.061 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:35:20.121349+0000 36'1 36:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:09:02.807588+0000 0 0 2024-08-18T21:36:02.061 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:35:22.401149+0000 36'3 36:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:37:45.466705+0000 0 0 2024-08-18T21:36:02.061 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:35:20.571486+0000 36'2 36:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:58:10.797373+0000 0 0 2024-08-18T21:36:02.061 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:35:30.624788+0000 36'4 36:62 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:57:59.352479+0000 1 0 2024-08-18T21:36:02.061 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 6 0 0 0 0 3435 0 0 7 0 7 active+clean 2024-08-18T21:35:25.958957+0000 36'7 36:55 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:53:10.798788+0000 0 0 2024-08-18T21:36:02.061 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:36:02.061 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:36:02.061 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:36:02.073 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-08-18T21:36:02.074 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:36:02.074 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:36:02.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:36:02.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:36:02.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:36:02.384 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:36:02.384 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-08-18T21:36:02.384 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-08-18T21:36:02.922 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already down. 2024-08-18T21:36:02.938 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-08-18T21:36:02.939 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-08-18T21:36:03.532 INFO:tasks.workunit.client.0.smithi099.stderr:osd.1 is already down. 2024-08-18T21:36:03.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-08-18T21:36:03.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-08-18T21:36:04.197 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already down. 2024-08-18T21:36:04.214 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-08-18T21:36:04.214 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.1 2024-08-18T21:36:04.810 INFO:tasks.workunit.client.0.smithi099.stderr:osd.1 is already out. 2024-08-18T21:36:04.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-08-18T21:36:04.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-08-18T21:36:05.415 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already out. 2024-08-18T21:36:05.433 INFO:tasks.workunit.client.0.smithi099.stdout:bringing up non_divergent 1 2 2024-08-18T21:36:05.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 1 2' 2024-08-18T21:36:05.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:36:05.865 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:36:05.865 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:24.992473+0000 36'3 36:51 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:43:18.899770+0000 0 0 2024-08-18T21:36:05.865 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:18.524452+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:22:51.247039+0000 0 0 2024-08-18T21:36:05.865 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-08-18T21:35:21.532043+0000 36'4 36:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:28:36.956355+0000 0 0 2024-08-18T21:36:05.865 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:25.594716+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T09:21:46.250195+0000 0 0 2024-08-18T21:36:05.865 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:29.535748+0000 36'3 36:61 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:19:51.726053+0000 1 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 0 0 0 3435 0 0 6 0 6 stale+active+clean 2024-08-18T21:35:28.529528+0000 36'6 36:64 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T03:41:16.961736+0000 2 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:33.539279+0000 36'3 36:61 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-20T09:25:31.831796+0000 0 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-08-18T21:35:31.655003+0000 36'2 36:60 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:48:14.349444+0000 1 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-08-18T21:35:27.561149+0000 36'4 36:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T01:57:12.845572+0000 0 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 stale+active+clean 2024-08-18T21:35:24.402639+0000 36'6 36:54 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:25:26.732185+0000 0 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-08-18T21:35:24.577380+0000 36'2 36:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:49:54.996068+0000 0 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 stale+active+clean 2024-08-18T21:35:19.084810+0000 36'5 36:63 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:23:43.334292+0000 0 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 stale+active+clean 2024-08-18T21:35:21.097125+0000 36'1 36:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:54:29.086185+0000 0 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:22.539811+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:38:14.898239+0000 0 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-08-18T21:35:20.431242+0000 36'2 36:50 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:22:05.930717+0000 0 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-08-18T21:35:26.929556+0000 36'2 36:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:16:28.438167+0000 0 0 2024-08-18T21:36:05.866 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:15.449055+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-19T23:58:49.929086+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:21.383299+0000 36'3 36:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:51:46.629110+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 0 0 0 1851 0 0 6 0 6 stale+active+clean 2024-08-18T21:35:32.549877+0000 36'6 36:64 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:40:28.398677+0000 2 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-08-18T21:35:23.406346+0000 36'4 36:52 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:37:27.621837+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:24.038460+0000 36'3 36:51 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:13:50.898681+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 2 0 0 0 0 1234 0 0 4 0 4 stale+active+clean 2024-08-18T21:35:22.093401+0000 36'4 36:52 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T05:03:13.803155+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 stale+active+clean 2024-08-18T21:35:23.048047+0000 36'1 36:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:57:11.387863+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:23.572198+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:43:59.581693+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:25.419256+0000 36'3 36:51 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:20:44.147140+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:26.643936+0000 36'3 36:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:16:18.870570+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-08-18T21:35:19.516597+0000 36'4 36:56 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:28:31.243352+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 stale+active+clean 2024-08-18T21:35:20.121349+0000 36'1 36:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:09:02.807588+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-08-18T21:35:22.401149+0000 36'3 36:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:37:45.466705+0000 0 0 2024-08-18T21:36:05.867 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-08-18T21:35:20.571486+0000 36'2 36:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:58:10.797373+0000 0 0 2024-08-18T21:36:05.868 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-08-18T21:35:30.624788+0000 36'4 36:62 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:57:59.352479+0000 1 0 2024-08-18T21:36:05.868 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 6 0 0 0 0 3435 0 0 7 0 7 stale+active+clean 2024-08-18T21:35:25.958957+0000 36'7 36:55 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:53:10.798788+0000 0 0 2024-08-18T21:36:05.868 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:36:05.868 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:36:05.868 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:36:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-08-18T21:36:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-08-18T21:36:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:36:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:36:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-18T21:36:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:36:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-18T21:36:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:36:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:36:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:36:05.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:36:05.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-18T21:36:05.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-18T21:36:05.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:36:05.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:36:05.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:36:05.885 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:36:05.885 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:36:05.885 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:36:05.885 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:36:05.885 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:36:05.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-18T21:36:05.887 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:36:05.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-18T21:36:05.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:36:05.888 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-18T21:36:05.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-18T21:36:05.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:36:05.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:36:05.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:36:05.922 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:36:05.925+0000 7fcc47a967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:36:05.924 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:36:05.929+0000 7fcc47a967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:36:05.925 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:36:05.929+0000 7fcc47a967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:36:06.309 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-18T21:36:06.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:36:06.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:36:06.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:36:06.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:36:06.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:36:06.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:36:06.310 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:36:06.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:36:06.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:36:06.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:36:07.194 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:36:07.197+0000 7fcc47a967c0 -1 Falling back to public interface 2024-08-18T21:36:07.759 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:36:07.759 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:36:07.760 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:36:07.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:36:07.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:36:07.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:36:08.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:36:09.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:36:09.236 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:36:09.236 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:36:09.237 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:36:09.237 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:36:09.237 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:36:09.683 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:36:10.653 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:36:10.657+0000 7fcc47a967c0 -1 osd.1 36 log_to_monitors true 2024-08-18T21:36:10.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:36:10.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:36:10.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:36:10.685 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:36:10.686 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:36:10.686 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:36:11.261 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up out weight 0 up_from 43 up_thru 34 down_at 38 last_clean_interval [10,36) [v2:127.0.0.1:6802/3809673106,v1:127.0.0.1:6803/3809673106] [v2:127.0.0.1:6804/3809673106,v1:127.0.0.1:6805/3809673106] exists,up d2abeb81-0f38-419c-ba93-2eea17b6fc91 2024-08-18T21:36:11.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:36:11.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:36:11.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:36:11.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-08-18T21:36:11.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-08-18T21:36:11.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:36:11.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:36:11.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-18T21:36:11.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:36:11.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-08-18T21:36:11.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:36:11.266 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:36:11.266 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:36:11.266 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:36:11.266 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-18T21:36:11.266 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-18T21:36:11.266 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:36:11.266 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:36:11.266 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:36:11.266 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:36:11.266 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:36:11.266 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:36:11.267 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:36:11.268 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-08-18T21:36:11.268 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-18T21:36:11.268 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:36:11.269 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:36:11.269 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-08-18T21:36:11.269 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-18T21:36:11.269 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:36:11.269 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:36:11.269 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:36:11.305 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:36:11.309+0000 7f6a2b60e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:36:11.308 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:36:11.313+0000 7f6a2b60e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:36:11.309 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:36:11.313+0000 7f6a2b60e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:36:11.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-18T21:36:11.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:36:11.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:36:11.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:36:11.806 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:36:11.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:36:11.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:36:11.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:36:11.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:36:11.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:36:12.365 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:36:12.554 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:36:12.557+0000 7f6a2b60e7c0 -1 Falling back to public interface 2024-08-18T21:36:13.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:36:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:36:13.368 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:36:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:36:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:36:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:36:13.858 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:36:14.862 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:36:14.862 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:36:14.862 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:36:14.863 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:36:14.863 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:36:14.863 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:36:15.340 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:36:15.864 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:36:15.869+0000 7f6a2b60e7c0 -1 osd.2 36 log_to_monitors true 2024-08-18T21:36:16.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:36:16.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:36:16.342 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:36:16.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:36:16.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:36:16.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:36:16.842 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up out weight 0 up_from 45 up_thru 34 down_at 39 last_clean_interval [15,36) [v2:127.0.0.1:6810/1808399366,v1:127.0.0.1:6811/1808399366] [v2:127.0.0.1:6812/1808399366,v1:127.0.0.1:6813/1808399366] exists,up 33c0de0c-1a9c-4bb2-9e7b-3a49e559e1e4 2024-08-18T21:36:16.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:36:16.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:36:16.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:36:16.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-08-18T21:36:16.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.1 2024-08-18T21:36:17.397 INFO:tasks.workunit.client.0.smithi099.stderr:osd.1 is already in. 2024-08-18T21:36:17.414 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-08-18T21:36:17.414 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-08-18T21:36:18.006 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already in. 2024-08-18T21:36:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-18T21:36:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-08-18T21:36:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:36:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:36:18.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:36:18.025 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-18T21:36:18.026 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:36:18.026 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:36:18.026 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:36:18.026 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:36:18.027 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:36:18.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-18T21:36:18.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:36:18.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:36:18.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:36:18.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:36:18.088 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:36:18.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:36:18.660 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:36:18.660 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:36:18.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:36:18.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:36:18.660 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:36:18.824 INFO:tasks.workunit.client.0.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-08-18T21:36:18.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq= 2024-08-18T21:36:18.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z '' 2024-08-18T21:36:18.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: continue 2024-08-18T21:36:18.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:36:18.826 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:36:19.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=184683593732 2024-08-18T21:36:19.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 184683593732 2024-08-18T21:36:19.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 1-184683593732' 2024-08-18T21:36:19.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:36:19.076 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:36:19.243 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=193273528323 2024-08-18T21:36:19.243 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 193273528323 2024-08-18T21:36:19.243 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 1-184683593732 2-193273528323' 2024-08-18T21:36:19.243 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:36:19.243 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-184683593732 2024-08-18T21:36:19.243 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:36:19.244 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:36:19.245 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-184683593732 2024-08-18T21:36:19.245 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:36:19.246 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=184683593732 2024-08-18T21:36:19.246 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 184683593732' 2024-08-18T21:36:19.246 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 184683593732 2024-08-18T21:36:19.247 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:36:19.683 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 184683593730 -lt 184683593732 2024-08-18T21:36:19.683 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:36:20.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:36:20.685 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:36:21.144 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 184683593732 -lt 184683593732 2024-08-18T21:36:21.144 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:36:21.145 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-193273528323 2024-08-18T21:36:21.145 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:36:21.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:36:21.148 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-193273528323 2024-08-18T21:36:21.148 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:36:21.149 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=193273528323 2024-08-18T21:36:21.149 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 193273528323 2024-08-18T21:36:21.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 193273528323' 2024-08-18T21:36:21.150 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:36:21.599 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 193273528323 -lt 193273528323 2024-08-18T21:36:21.600 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:36:21.601 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:36:21.601 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:36:22.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-08-18T21:36:22.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:36:22.179 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:36:22.179 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:36:22.179 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:36:22.180 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:36:22.180 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:36:22.180 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:36:22.635 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:36:22.635 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:36:22.635 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:36:22.635 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:36:23.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:36:23.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-18T21:36:23.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-18T21:36:23.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-18T21:36:23.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:36:23.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-18T21:36:23.299 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:36:23.299 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:36:23.300 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:36:23.300 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:36:23.300 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:36:23.300 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:36:23.301 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:36:23.301 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:36:23.718 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:36:23.719 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:36:23.719 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:36:23.719 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:36:24.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:36:24.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:36:24.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:36:24.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:36:24.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:36:24.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:36:24.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:36:24.284 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:36:24.284 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:36:24.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:36:24.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:36:24.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-18T21:36:24.846 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:36:24.846 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-18T21:36:25.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:36:25.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:36:25.047 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:36:25.047 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:36:25.047 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:36:25.048 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:36:25.048 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:36:25.048 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:36:25.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:36:25.484 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:36:25.485 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:36:25.485 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:36:26.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:36:26.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:36:26.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:36:26.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:36:26.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:36:26.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:36:26.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:36:26.035 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:36:26.035 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:36:26.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:36:26.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:36:26.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-18T21:36:26.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:36:26.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-18T21:36:27.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:36:27.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:36:27.018 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:36:27.018 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:36:27.018 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:36:27.018 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:36:27.019 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:36:27.019 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:36:27.461 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:36:27.462 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:36:27.462 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:36:27.462 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:36:28.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:36:28.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:36:28.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:36:28.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:36:28.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:36:28.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:36:28.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:36:28.046 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:36:28.046 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:36:28.595 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:36:28.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:36:28.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-18T21:36:28.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:36:28.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-18T21:36:29.397 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:36:29.397 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:36:29.398 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:36:29.398 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:36:29.398 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:36:29.398 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:36:29.399 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:36:29.399 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:36:29.847 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:36:29.848 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:36:29.848 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:36:29.848 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:36:30.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:36:30.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:36:30.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:36:30.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:36:30.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:36:30.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:36:30.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:36:30.407 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:36:30.407 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:36:30.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:36:30.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:36:30.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-18T21:36:30.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:36:30.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-18T21:36:32.586 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:36:32.586 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:36:32.588 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:36:32.588 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:36:32.588 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:36:32.589 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:36:32.589 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:36:32.589 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:36:33.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:36:33.014 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:36:33.015 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:36:33.015 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:36:33.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:36:33.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:36:33.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:36:33.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:36:33.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:36:33.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:36:33.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:36:33.563 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:36:33.563 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:36:34.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:36:34.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:36:34.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-18T21:36:34.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:36:34.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-18T21:36:37.366 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:36:37.366 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:36:37.367 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:36:37.367 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:36:37.367 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:36:37.367 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:36:37.368 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:36:37.368 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:36:37.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:36:37.808 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:36:37.808 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:36:37.809 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:36:38.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:36:38.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:36:38.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:36:38.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:36:38.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:36:38.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:36:38.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:36:38.382 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:36:38.382 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:36:38.941 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:36:38.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:36:38.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-18T21:36:38.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:36:38.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-18T21:36:45.343 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:36:45.343 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:36:45.344 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:36:45.344 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:36:45.344 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:36:45.344 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:36:45.345 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:36:45.345 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:36:45.789 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:36:45.790 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:36:45.790 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:36:45.790 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:36:46.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:36:46.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:36:46.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:36:46.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:36:46.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:36:46.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:36:46.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:36:46.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:36:46.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:36:46.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:36:46.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:36:46.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-18T21:36:46.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:36:46.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-18T21:36:54.212 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:36:54.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:36:54.213 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:36:54.213 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:36:54.213 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:36:54.214 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:36:54.214 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:36:54.214 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:36:54.681 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:36:54.682 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:36:54.682 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:36:54.682 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:36:55.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:36:55.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:36:55.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:36:55.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:36:55.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:36:55.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:36:55.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:36:55.260 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:36:55.260 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:36:55.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:36:55.823 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:36:55.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-18T21:36:55.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-18T21:36:56.362 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:36:56.362 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_fingerprint": "c80ebbcd-42be-47a7-8ff8-155f2ae99129", 2024-08-18T21:36:56.362 INFO:tasks.workunit.client.0.smithi099.stdout: "version": "19.0.0-5847-g79766939", 2024-08-18T21:36:56.362 INFO:tasks.workunit.client.0.smithi099.stdout: "commit": "797669399b32d2915e64eb6d4b1257ae3c50f80b", 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "timestamp": "2024-08-18T21:36:56.364962+0000", 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "tag": "", 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "health": { 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "status": "HEALTH_WARN", 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "checks": { 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "OSDMAP_FLAGS": { 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 17 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [], 2024-08-18T21:36:56.363 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: "OSD_DOWN": { 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 osds down", 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "osd.0 (root=default,host=smithi099) is down" 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.364 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.365 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:36:56.365 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.365 INFO:tasks.workunit.client.0.smithi099.stdout: "PG_DEGRADED": { 2024-08-18T21:36:56.365 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:36:56.365 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:36:56.365 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2024-08-18T21:36:56.365 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 32 2024-08-18T21:36:56.365 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.365 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:36:56.365 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.366 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.366 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.366 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.366 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,1]" 2024-08-18T21:36:56.366 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.366 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.366 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.366 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.366 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.366 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.366 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,1]" 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.367 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.a is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.b is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.c is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.368 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,1]" 2024-08-18T21:36:56.369 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.369 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.369 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.e is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.369 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.369 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.369 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,1]" 2024-08-18T21:36:56.369 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.369 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [2,1]" 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.370 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [2,1]" 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.371 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,1]" 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.372 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [1,2]" 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,1]" 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.373 INFO:tasks.workunit.client.0.smithi099.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "application not enabled on pool 'test'" 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-18T21:36:56.374 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: "mutes": [] 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_first_committed": 1, 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_last_committed": 1, 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap": { 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:36:56.375 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:33:05.203702Z", 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:33:05.203702Z", 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release": 19, 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release_name": "squid", 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "election_strategy": 1, 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "disallowed_leaders: ": "", 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": false, 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "tiebreaker_mon": "", 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_ranks: ": "", 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "features": { 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "persistent": [ 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "kraken", 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "luminous", 2024-08-18T21:36:56.376 INFO:tasks.workunit.client.0.smithi099.stdout: "mimic", 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap-prune", 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: "nautilus", 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: "octopus", 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: "pacific", 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: "elector-pinging", 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: "quincy", 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: "reef", 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: "squid" 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: "optional": [] 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: "mons": [ 2024-08-18T21:36:56.377 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: "rank": 0, 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "a", 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115", 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 0 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115/0", 2024-08-18T21:36:56.378 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: "priority": 0, 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 0, 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_location": "{}" 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: "quorum": [ 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: 0 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap": { 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 50, 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:36:56.379 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:33:06.219529+0000", 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:36:18.535300+0000", 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "last_up_change": "2024-08-18T21:36:16.188813+0000", 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "last_in_change": "2024-08-18T21:36:17.828016+0000", 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_num": 5800384, 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_set": [ 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "nodown", 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "noin", 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "noout", 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "pglog_hardlimit", 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snapdirs", 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "recovery_deletes", 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: "sortbitwise" 2024-08-18T21:36:56.380 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_version": 9, 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "full_ratio": 0.99000000953674316, 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_snapshot": "", 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_max": 1, 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "max_osd": 3, 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "require_min_compat_client": "luminous", 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "min_compat_client": "jewel", 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "require_osd_release": "squid", 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "allow_crimson": false, 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: "pools": [ 2024-08-18T21:36:56.381 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_name": "test", 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "create_time": "2024-08-18T21:35:03.265323+0000", 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": 1, 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_names": "hashpspool", 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "size": 3, 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "min_size": 2, 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_rule": 0, 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-08-18T21:36:56.382 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "is_stretch_pool": false, 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "object_hash": 2, 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_autoscale_mode": "on", 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num": 32, 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num": 32, 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num_target": 32, 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_target": 32, 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_pending": 32, 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pg_merge_meta": { 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "source_pgid": "0.0", 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "ready_epoch": 0, 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_started": 0, 2024-08-18T21:36:56.383 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": 0, 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "source_version": "0'0", 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "target_version": "0'0" 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "last_change": "34", 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend": "0", 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_preluminous": "32", 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "auid": 0, 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_mode": "selfmanaged", 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_seq": 0, 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_epoch": 0, 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_snaps": [], 2024-08-18T21:36:56.384 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps": "[]", 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_bytes": 0, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_objects": 0, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "tiers": [], 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "tier_of": -1, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "read_tier": -1, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "write_tier": -1, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_mode": "none", 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_bytes": 0, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_objects": 0, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_flush_age": 0, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_evict_age": 0, 2024-08-18T21:36:56.385 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profile": "", 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_params": { 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "none" 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_period": 0, 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_count": 0, 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: "use_gmt_hitset": true, 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_search_last_n": 0, 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: "grade_table": [], 2024-08-18T21:36:56.386 INFO:tasks.workunit.client.0.smithi099.stdout: "stripe_width": 0, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "expected_num_objects": 0, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "fast_read": false, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "options": {}, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "application_metadata": {}, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "read_balance": { 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "score_type": "Fair distribution", 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "score_acting": 2.2400000095367432, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "score_stable": 2.2400000095367432, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_score": 0.67000001668930054, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "raw_score_acting": 1.5, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "raw_score_stable": 1.5, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity_weighted": 1, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "average_primary_affinity": 1, 2024-08-18T21:36:56.387 INFO:tasks.workunit.client.0.smithi099.stdout: "average_primary_affinity_weighted": 1 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: "osds": [ 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "6e2c559a-40e6-4a8a-8afe-cf8a54df7cbc", 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 0, 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 0, 2024-08-18T21:36:56.388 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 0, 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 5, 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 34, 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 37, 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6802", 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.389 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6803", 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6804", 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:36:56.390 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.391 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.391 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.391 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6805", 2024-08-18T21:36:56.391 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:36:56.391 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.391 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.391 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.391 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:36:56.391 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.391 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6808", 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6809", 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.392 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6806", 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6807", 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6803/2216089247", 2024-08-18T21:36:56.393 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6805/2216089247", 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2216089247", 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2216089247", 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: "exists" 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "d2abeb81-0f38-419c-ba93-2eea17b6fc91", 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:36:56.394 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 10, 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 36, 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 43, 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 48, 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 38, 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6802", 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:36:56.395 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6803", 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6804", 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:36:56.396 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6805", 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.397 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6808", 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6809", 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.398 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6806", 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6807", 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6803/3809673106", 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6805/3809673106", 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3809673106", 2024-08-18T21:36:56.399 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3809673106", 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "33c0de0c-1a9c-4bb2-9e7b-3a49e559e1e4", 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:36:56.400 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 15, 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 36, 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 45, 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 48, 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 39, 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:36:56.401 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:36:56.402 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.403 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:36:56.404 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/1808399366", 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/1808399366", 2024-08-18T21:36:56.405 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1808399366", 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1808399366", 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_xinfo": [ 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:36:02.754068+0000", 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:36:56.406 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:33:42.466017+0000", 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:36:03.339262+0000", 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:36:56.407 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:34:20.595763+0000", 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:36:03.968322+0000", 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:34:59.122164+0000", 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap": [], 2024-08-18T21:36:56.408 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_items": [], 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_primaries": [], 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_temp": [], 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_temp": [], 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "blocklist": {}, 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "range_blocklist": {}, 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profiles": { 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "default": { 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "k": "2", 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "m": "2", 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "plugin": "jerasure", 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "technique": "reed_sol_van" 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.409 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps_queue": [], 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: "new_removed_snaps": [], 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: "new_purged_snaps": [], 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_node_flags": {}, 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: "device_class_flags": {}, 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": { 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_enabled": false, 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_bucket_count": 0, 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: "degraded_stretch_mode": 0, 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: "recovering_stretch_mode": 0, 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_bucket": 0 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.410 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_metadata": [ 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6804/2216089247,v1:127.0.0.1:6805/2216089247]", 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:36:56.411 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:36:56.412 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:33:13.175443Z", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6802/2216089247,v1:127.0.0.1:6803/2216089247]", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2216089247,v1:127.0.0.1:6809/2216089247]", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2216089247,v1:127.0.0.1:6807/2216089247]", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:36:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/0", 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.414 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6804/3809673106,v1:127.0.0.1:6805/3809673106]", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:36:56.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:33:50.010420Z", 2024-08-18T21:36:56.416 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6802/3809673106,v1:127.0.0.1:6803/3809673106]", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3809673106,v1:127.0.0.1:6809/3809673106]", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3809673106,v1:127.0.0.1:6807/3809673106]", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:36:56.417 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/1", 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.418 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/1808399366,v1:127.0.0.1:6813/1808399366]", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:36:56.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:34:28.689269Z", 2024-08-18T21:36:56.420 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/1808399366,v1:127.0.0.1:6811/1808399366]", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1808399366,v1:127.0.0.1:6817/1808399366]", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1808399366,v1:127.0.0.1:6815/1808399366]", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:36:56.421 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/2", 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_clean_epochs": { 2024-08-18T21:36:56.422 INFO:tasks.workunit.client.0.smithi099.stdout: "min_last_epoch_clean": 0, 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": { 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: "per_pool": [ 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: "poolid": 1, 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: "floor": 33 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_epochs": [ 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 43 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.423 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 45 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_first_committed": 1, 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_last_committed": 50, 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: "crushmap": { 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": [ 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.0", 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:36:56.424 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.1", 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.2", 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: "types": [ 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.425 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 0, 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd" 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "host" 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 2, 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "chassis" 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 3, 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "rack" 2024-08-18T21:36:56.426 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 4, 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "row" 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 5, 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pdu" 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 6, 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pod" 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.427 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 7, 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "room" 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 8, 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "datacenter" 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 9, 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "zone" 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 10, 2024-08-18T21:36:56.428 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "region" 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "root" 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: "buckets": [ 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -1, 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default", 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:36:56.429 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -2, 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default~hdd", 2024-08-18T21:36:56.430 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.431 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.432 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:36:56.432 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099", 2024-08-18T21:36:56.432 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:36:56.432 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:36:56.432 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:36:56.432 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:36:56.432 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:36:56.432 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:36:56.432 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:36:56.433 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099~hdd", 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:36:56.434 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.435 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: "rules": [ 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_id": 0, 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_name": "replicated_rule", 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: "steps": [ 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "take", 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: "item": -1, 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: "item_name": "default" 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.436 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "choose_firstn", 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 0, 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "osd" 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "emit" 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: "tunables": { 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_tries": 0, 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_fallback_tries": 0, 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_total_tries": 50, 2024-08-18T21:36:56.437 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_descend_once": 1, 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_vary_r": 1, 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_stable": 1, 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_descents": 100, 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_collision_tries": 100, 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "straw_calc_version": 1, 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "allowed_bucket_algs": 54, 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "profile": "jewel", 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_tunables": 1, 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "legacy_tunables": 0, 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "minimum_required_version": "jewel", 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables": 1, 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables2": 1, 2024-08-18T21:36:56.438 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v2_rules": 0, 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables3": 1, 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v3_rules": 0, 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v4_buckets": 1, 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables5": 1, 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v5_rules": 0, 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: "has_msr_rules": 0 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_args": {} 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: "fsmap": { 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: "btime": "2024-08-18T21:33:06:194092+0000", 2024-08-18T21:36:56.439 INFO:tasks.workunit.client.0.smithi099.stdout: "default_fscid": -1, 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": { 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": {}, 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "ro_compat": {}, 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "incompat": { 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_1": "base v0.20", 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_2": "client writeable ranges", 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_3": "default file layouts on dirs", 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_4": "dir inode in separate object", 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_5": "mds uses versioned encoding", 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_8": "no anchor table", 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_9": "file layout v2", 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_10": "snaprealm v2", 2024-08-18T21:36:56.440 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_11": "minor log segments", 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_12": "quiesce subvolumes" 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_flags": { 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: "enable_multiple": true, 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: "ever_enabled_multiple": true 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: "standbys": [], 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: "filesystems": [] 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_first_committed": 1, 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_last_committed": 1, 2024-08-18T21:36:56.441 INFO:tasks.workunit.client.0.smithi099.stdout: "auth": { 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 5, 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "num_secrets": 3 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 32, 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_active": 32, 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_unknown": 0, 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osd": 3, 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_sum": { 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 61700, 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 100, 2024-08-18T21:36:56.442 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 0, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 300, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 100, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 100, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 93, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 93, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:36:56.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:36:56.444 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:36:56.445 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 0, 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 0, 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 100, 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 100, 2024-08-18T21:36:56.446 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 64, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 64, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 0 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum": { 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 96, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83848, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2304, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:36:56.447 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314488952, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036686848, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2359296, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 534432, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10392, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416936 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:36:56.448 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.449 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum_by_class": { 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "hdd": { 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 96, 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83848, 2024-08-18T21:36:56.450 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2304, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314488952, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036686848, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2359296, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 534432, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10392, 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416936 2024-08-18T21:36:56.451 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:36:56.452 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:36:56.453 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.453 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:36:56.453 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:36:56.453 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.453 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.453 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_stats": [ 2024-08-18T21:36:56.453 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.453 INFO:tasks.workunit.client.0.smithi099.stdout: "poolid": 1, 2024-08-18T21:36:56.453 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 32, 2024-08-18T21:36:56.453 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:36:56.453 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 61700, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 100, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 0, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 300, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 100, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 100, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:36:56.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 93, 2024-08-18T21:36:56.455 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 93, 2024-08-18T21:36:56.455 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:36:56.455 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:36:56.455 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:36:56.455 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:36:56.455 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:36:56.455 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:36:56.455 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:36:56.455 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:36:56.455 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:36:56.455 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:36:56.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 1220608, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 184566, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:36:56.457 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 100, 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 100, 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 64, 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 64, 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 3 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_stats": [ 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 21474836520 2024-08-18T21:36:56.458 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 184683593740 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 193273528330 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_state": [ 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: "state": "active+undersized+degraded", 2024-08-18T21:36:56.459 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 32 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_osd": [ 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 8, 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 32, 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:36:56.460 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 0, 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 24, 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 32, 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": [ 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": {} 2024-08-18T21:36:56.461 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "servicemap": { 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 5, 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:36:01.890115+0000", 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "services": {} 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_first_committed": 1, 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_last_committed": 104, 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_first_committed": 1, 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_last_committed": 428, 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "paxos": { 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 584, 2024-08-18T21:36:56.462 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pn": 0, 2024-08-18T21:36:56.463 INFO:tasks.workunit.client.0.smithi099.stdout: "accepted_pn": 0 2024-08-18T21:36:56.463 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:36:56.463 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:36:56.463 INFO:tasks.workunit.client.0.smithi099.stderr:report 1973983612 2024-08-18T21:36:56.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-18T21:36:56.464 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-08-18T21:36:56.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-08-18T21:36:56.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-08-18T21:36:56.464 INFO:tasks.workunit.client.0.smithi099.stdout:writing non-divergent object existing_4 2024-08-18T21:36:56.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:36:56.798 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:36:56.798 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060700+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:49:40.518308+0000 0 0 2024-08-18T21:36:56.798 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.792738+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:10:03.086555+0000 0 0 2024-08-18T21:36:56.798 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.792672+0000 36'4 50:68 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:44:42.279447+0000 0 0 2024-08-18T21:36:56.798 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.794552+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:32:35.017116+0000 0 0 2024-08-18T21:36:56.798 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793126+0000 36'3 50:73 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:04:53.673735+0000 1 0 2024-08-18T21:36:56.798 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.793821+0000 36'5 50:75 [1,2] 1 [1,2] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:40:43.447167+0000 2 0 2024-08-18T21:36:56.798 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793178+0000 36'3 50:73 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:53:38.202575+0000 0 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.796571+0000 36'2 50:72 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:17:19.819950+0000 1 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.792482+0000 36'4 50:68 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:01:15.594403+0000 0 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-18T21:36:19.060214+0000 36'6 50:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:27:39.408296+0000 0 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.793062+0000 36'2 50:66 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-20T07:58:37.646145+0000 0 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.678911+0000 36'5 50:75 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:24:52.944633+0000 0 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.792497+0000 36'1 50:61 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:12:27.011984+0000 0 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.796210+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:19:03.521863+0000 0 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:19.060000+0000 36'2 50:59 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:38:20.186954+0000 0 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.795623+0000 36'2 50:62 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:11:33.117016+0000 0 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.792399+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:42:41.761298+0000 0 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.059997+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:26:56.103631+0000 0 0 2024-08-18T21:36:56.799 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.792213+0000 36'4 50:74 [1,2] 1 [1,2] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:03:06.414446+0000 2 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:19.060940+0000 36'4 50:61 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:16:08.923652+0000 0 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060354+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:09:10.020078+0000 0 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.795482+0000 36'3 50:63 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T08:44:39.434197+0000 0 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.796728+0000 36'1 50:61 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:45:38.775344+0000 0 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793454+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:16:56.012624+0000 0 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.062198+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:06:12.002363+0000 0 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.678901+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:56:17.057598+0000 0 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.678909+0000 36'4 50:68 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:27:51.072853+0000 0 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.794223+0000 36'1 50:61 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:55:35.538743+0000 0 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060250+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:18:42.255563+0000 0 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.794083+0000 36'2 50:66 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:16:51.117645+0000 0 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.793596+0000 36'4 50:74 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:20:45.081929+0000 1 0 2024-08-18T21:36:56.800 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.795883+0000 36'5 50:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:31:12.592149+0000 0 0 2024-08-18T21:36:56.801 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:36:56.801 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:36:56.801 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:36:56.813 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-08-18T21:36:56.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-18T21:36:56.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-08-18T21:36:56.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:36:56.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:36:56.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:36:56.913 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-18T21:36:56.913 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:36:56.913 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:36:56.913 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:36:56.914 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:36:56.914 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:36:56.982 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-18T21:36:56.982 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:36:56.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:36:56.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:36:56.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:36:56.983 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:36:57.480 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:36:57.480 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:36:57.480 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:36:57.480 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:36:57.480 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:36:57.481 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:36:57.614 INFO:tasks.workunit.client.0.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-08-18T21:36:57.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq= 2024-08-18T21:36:57.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z '' 2024-08-18T21:36:57.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: continue 2024-08-18T21:36:57.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:36:57.616 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:36:57.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=184683593742 2024-08-18T21:36:57.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 184683593742 2024-08-18T21:36:57.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 1-184683593742' 2024-08-18T21:36:57.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:36:57.779 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:36:57.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=193273528333 2024-08-18T21:36:57.937 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 193273528333 2024-08-18T21:36:57.937 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 1-184683593742 2-193273528333' 2024-08-18T21:36:57.937 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:36:57.937 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-184683593742 2024-08-18T21:36:57.937 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:36:57.939 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:36:57.939 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-184683593742 2024-08-18T21:36:57.939 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:36:57.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=184683593742 2024-08-18T21:36:57.941 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 184683593742 2024-08-18T21:36:57.941 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 184683593742' 2024-08-18T21:36:57.941 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:36:58.383 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 184683593742 -lt 184683593742 2024-08-18T21:36:58.383 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:36:58.383 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-193273528333 2024-08-18T21:36:58.384 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:36:58.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:36:58.386 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-193273528333 2024-08-18T21:36:58.386 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:36:58.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=193273528333 2024-08-18T21:36:58.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 193273528333' 2024-08-18T21:36:58.387 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 193273528333 2024-08-18T21:36:58.387 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:36:58.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 193273528331 -lt 193273528333 2024-08-18T21:36:58.910 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:36:59.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:36:59.912 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:37:00.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 193273528333 -lt 193273528333 2024-08-18T21:37:00.367 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:37:00.368 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:37:00.368 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:37:00.947 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-08-18T21:37:00.947 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:37:00.948 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:37:00.948 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:37:00.949 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:37:00.949 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:37:00.949 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:37:00.949 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:37:01.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:37:01.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:37:01.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:37:01.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:37:01.928 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:37:01.928 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-18T21:37:01.928 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-18T21:37:01.928 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-18T21:37:01.928 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:37:01.928 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-18T21:37:02.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:37:02.029 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:37:02.029 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:37:02.029 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:37:02.030 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:37:02.030 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:37:02.030 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:37:02.030 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:37:02.476 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:37:02.477 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:37:02.477 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:37:02.477 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:37:03.032 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:37:03.032 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:37:03.032 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:37:03.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:37:03.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:37:03.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:37:03.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:37:03.033 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:37:03.033 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:37:03.632 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:37:03.645 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:37:03.645 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-18T21:37:03.645 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:37:03.645 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-18T21:37:03.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:37:03.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:37:03.834 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:37:03.834 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:37:03.834 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:37:03.834 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:37:03.834 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:37:03.834 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:37:04.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:37:04.276 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:37:04.276 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:37:04.276 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:37:04.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:37:04.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:37:04.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:37:04.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:37:04.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:37:04.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:37:04.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:37:04.833 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:37:04.834 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:37:05.420 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:37:05.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:37:05.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-18T21:37:05.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:37:05.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-18T21:37:05.822 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:37:05.822 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:37:05.823 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:37:05.823 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:37:05.823 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:37:05.823 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:37:05.823 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:37:05.824 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:37:06.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:37:06.260 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:37:06.261 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:37:06.261 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:37:06.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:37:06.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:37:06.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:37:06.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:37:06.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:37:06.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:37:06.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:37:06.825 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:37:06.825 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:37:07.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:37:07.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:37:07.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-18T21:37:07.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:37:07.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-18T21:37:08.232 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:37:08.233 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:37:08.234 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:37:08.234 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:37:08.234 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:37:08.234 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:37:08.234 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:37:08.234 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:37:08.670 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:37:08.670 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:37:08.670 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:37:08.670 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:37:09.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:37:09.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:37:09.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:37:09.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:37:09.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:37:09.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:37:09.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:37:09.248 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:37:09.248 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:37:09.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:37:09.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:37:09.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-18T21:37:09.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:37:09.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-18T21:37:11.397 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:37:11.397 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:37:11.398 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:37:11.398 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:37:11.398 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:37:11.398 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:37:11.399 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:37:11.399 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:37:11.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:37:11.843 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:37:11.843 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:37:11.844 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:37:12.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:37:12.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:37:12.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:37:12.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:37:12.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:37:12.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:37:12.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:37:12.413 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:37:12.413 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:37:12.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:37:12.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:37:12.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-18T21:37:12.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:37:12.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-18T21:37:16.185 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:37:16.185 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:37:16.186 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:37:16.186 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:37:16.186 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:37:16.186 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:37:16.187 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:37:16.187 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:37:16.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:37:16.677 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:37:16.677 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:37:16.677 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:37:17.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:37:17.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:37:17.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:37:17.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:37:17.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:37:17.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:37:17.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:37:17.236 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:37:17.236 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:37:17.775 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:37:17.775 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:37:17.776 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-18T21:37:17.776 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:37:17.776 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-18T21:37:24.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:37:24.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:37:24.178 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:37:24.178 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:37:24.178 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:37:24.178 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:37:24.178 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:37:24.178 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:37:24.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:37:24.623 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:37:24.623 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:37:24.624 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:37:25.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:37:25.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:37:25.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:37:25.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:37:25.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:37:25.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:37:25.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:37:25.213 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:37:25.214 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:37:25.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:37:25.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:37:25.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-18T21:37:25.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:37:25.769 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-18T21:37:33.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:37:33.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:37:33.070 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:37:33.071 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:37:33.071 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:37:33.071 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:37:33.071 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:37:33.071 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:37:33.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:37:33.508 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:37:33.508 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:37:33.508 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:37:34.082 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-08-18T21:37:34.082 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:37:34.082 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:37:34.082 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:37:34.082 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:37:34.083 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:37:34.083 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:37:34.083 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:37:34.083 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:37:34.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:37:34.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:37:34.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-18T21:37:34.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-18T21:37:35.210 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:37:35.210 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_fingerprint": "c80ebbcd-42be-47a7-8ff8-155f2ae99129", 2024-08-18T21:37:35.211 INFO:tasks.workunit.client.0.smithi099.stdout: "version": "19.0.0-5847-g79766939", 2024-08-18T21:37:35.211 INFO:tasks.workunit.client.0.smithi099.stdout: "commit": "797669399b32d2915e64eb6d4b1257ae3c50f80b", 2024-08-18T21:37:35.211 INFO:tasks.workunit.client.0.smithi099.stdout: "timestamp": "2024-08-18T21:37:35.212748+0000", 2024-08-18T21:37:35.211 INFO:tasks.workunit.client.0.smithi099.stdout: "tag": "", 2024-08-18T21:37:35.211 INFO:tasks.workunit.client.0.smithi099.stdout: "health": { 2024-08-18T21:37:35.211 INFO:tasks.workunit.client.0.smithi099.stdout: "status": "HEALTH_WARN", 2024-08-18T21:37:35.212 INFO:tasks.workunit.client.0.smithi099.stdout: "checks": { 2024-08-18T21:37:35.212 INFO:tasks.workunit.client.0.smithi099.stdout: "OSDMAP_FLAGS": { 2024-08-18T21:37:35.212 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:37:35.212 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:37:35.212 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-18T21:37:35.212 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 17 2024-08-18T21:37:35.213 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.213 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [], 2024-08-18T21:37:35.213 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:37:35.213 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.213 INFO:tasks.workunit.client.0.smithi099.stdout: "OSD_DOWN": { 2024-08-18T21:37:35.213 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:37:35.214 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:37:35.214 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 osds down", 2024-08-18T21:37:35.214 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:37:35.214 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.214 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:37:35.215 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.215 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "osd.0 (root=default,host=smithi099) is down" 2024-08-18T21:37:35.215 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.215 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.215 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:37:35.215 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.216 INFO:tasks.workunit.client.0.smithi099.stdout: "PG_DEGRADED": { 2024-08-18T21:37:35.216 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:37:35.216 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:37:35.216 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-08-18T21:37:35.216 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 64 2024-08-18T21:37:35.216 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.217 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:37:35.217 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.217 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.0 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.217 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.217 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.218 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,1]" 2024-08-18T21:37:35.218 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.218 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.218 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.2 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.218 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.218 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.218 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.3 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.4 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.5 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,1]" 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.6 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.7 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.219 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.8 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.9 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.a is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.b is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.220 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.c is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.d is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,1]" 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.e is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.f is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,1]" 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.221 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.10 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,1]" 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.11 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.12 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.13 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.222 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.14 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,1]" 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.15 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.16 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.17 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.18 is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,1]" 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.223 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.19 is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1a is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1b is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1c is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.224 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1d is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1e is stuck undersized for 75s, current state active+undersized+degraded, last acting [1,2]" 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.1f is stuck undersized for 75s, current state active+undersized+degraded, last acting [2,1]" 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:37:35.225 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "application not enabled on pool 'test'" 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.226 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "mutes": [] 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_first_committed": 1, 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_last_committed": 1, 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap": { 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:33:05.203702Z", 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:33:05.203702Z", 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release": 19, 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release_name": "squid", 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "election_strategy": 1, 2024-08-18T21:37:35.227 INFO:tasks.workunit.client.0.smithi099.stdout: "disallowed_leaders: ": "", 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": false, 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "tiebreaker_mon": "", 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_ranks: ": "", 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "features": { 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "persistent": [ 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "kraken", 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "luminous", 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "mimic", 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap-prune", 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "nautilus", 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "octopus", 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "pacific", 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "elector-pinging", 2024-08-18T21:37:35.228 INFO:tasks.workunit.client.0.smithi099.stdout: "quincy", 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: "reef", 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: "squid" 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: "optional": [] 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: "mons": [ 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: "rank": 0, 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "a", 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.229 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115", 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 0 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115/0", 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: "priority": 0, 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 0, 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_location": "{}" 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.230 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: "quorum": [ 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: 0 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap": { 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 50, 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:33:06.219529+0000", 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:36:18.535300+0000", 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: "last_up_change": "2024-08-18T21:36:16.188813+0000", 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: "last_in_change": "2024-08-18T21:36:17.828016+0000", 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_num": 5800384, 2024-08-18T21:37:35.231 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_set": [ 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "nodown", 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "noin", 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "noout", 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "pglog_hardlimit", 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snapdirs", 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "recovery_deletes", 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "sortbitwise" 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_version": 9, 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "full_ratio": 0.99000000953674316, 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_snapshot": "", 2024-08-18T21:37:35.232 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_max": 1, 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "max_osd": 3, 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "require_min_compat_client": "luminous", 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "min_compat_client": "jewel", 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "require_osd_release": "squid", 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "allow_crimson": false, 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "pools": [ 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_name": "test", 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "create_time": "2024-08-18T21:35:03.265323+0000", 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": 1, 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_names": "hashpspool", 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:37:35.233 INFO:tasks.workunit.client.0.smithi099.stdout: "size": 3, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "min_size": 2, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_rule": 0, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "is_stretch_pool": false, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "object_hash": 2, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_autoscale_mode": "on", 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num": 32, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num": 32, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num_target": 32, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_target": 32, 2024-08-18T21:37:35.234 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_pending": 32, 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pg_merge_meta": { 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "source_pgid": "0.0", 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "ready_epoch": 0, 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_started": 0, 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": 0, 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "source_version": "0'0", 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "target_version": "0'0" 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "last_change": "34", 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend": "0", 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_preluminous": "32", 2024-08-18T21:37:35.235 INFO:tasks.workunit.client.0.smithi099.stdout: "auid": 0, 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_mode": "selfmanaged", 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_seq": 0, 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_epoch": 0, 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_snaps": [], 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps": "[]", 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_bytes": 0, 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_objects": 0, 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "tiers": [], 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "tier_of": -1, 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "read_tier": -1, 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "write_tier": -1, 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_mode": "none", 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_bytes": 0, 2024-08-18T21:37:35.236 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_objects": 0, 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_flush_age": 0, 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_evict_age": 0, 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profile": "", 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_params": { 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "none" 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_period": 0, 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_count": 0, 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "use_gmt_hitset": true, 2024-08-18T21:37:35.237 INFO:tasks.workunit.client.0.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_search_last_n": 0, 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "grade_table": [], 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "stripe_width": 0, 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "expected_num_objects": 0, 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "fast_read": false, 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "options": {}, 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "application_metadata": {}, 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "read_balance": { 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "score_type": "Fair distribution", 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "score_acting": 2.2400000095367432, 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "score_stable": 2.2400000095367432, 2024-08-18T21:37:35.238 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_score": 0.67000001668930054, 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: "raw_score_acting": 1.5, 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: "raw_score_stable": 1.5, 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity_weighted": 1, 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: "average_primary_affinity": 1, 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: "average_primary_affinity_weighted": 1 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: "osds": [ 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "6e2c559a-40e6-4a8a-8afe-cf8a54df7cbc", 2024-08-18T21:37:35.239 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 0, 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 0, 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 0, 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 5, 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 34, 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 37, 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.240 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6802", 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6803", 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.241 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6804", 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6805", 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.242 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6808", 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6809", 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.243 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6806", 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6807", 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2216089247 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6803/2216089247", 2024-08-18T21:37:35.244 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6805/2216089247", 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2216089247", 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2216089247", 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: "exists" 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "d2abeb81-0f38-419c-ba93-2eea17b6fc91", 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:37:35.245 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 10, 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 36, 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 43, 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 48, 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 38, 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6802", 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.246 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6803", 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6804", 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.247 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6805", 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6808", 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:37:35.248 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6809", 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6806", 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:37:35.249 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6807", 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 3809673106 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6803/3809673106", 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6805/3809673106", 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3809673106", 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3809673106", 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:37:35.250 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "33c0de0c-1a9c-4bb2-9e7b-3a49e559e1e4", 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 15, 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 36, 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 45, 2024-08-18T21:37:35.251 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 48, 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 39, 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.252 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.253 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.254 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1808399366 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.255 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/1808399366", 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/1808399366", 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1808399366", 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1808399366", 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_xinfo": [ 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.256 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:36:02.754068+0000", 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:33:42.466017+0000", 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:36:03.339262+0000", 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:37:35.257 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:34:20.595763+0000", 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:36:03.968322+0000", 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:34:59.122164+0000", 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:37:35.258 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap": [], 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_items": [], 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_primaries": [], 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_temp": [], 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_temp": [], 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "blocklist": {}, 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "range_blocklist": {}, 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profiles": { 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "default": { 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "k": "2", 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "m": "2", 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "plugin": "jerasure", 2024-08-18T21:37:35.259 INFO:tasks.workunit.client.0.smithi099.stdout: "technique": "reed_sol_van" 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps_queue": [], 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: "new_removed_snaps": [], 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: "new_purged_snaps": [], 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_node_flags": {}, 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: "device_class_flags": {}, 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": { 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_enabled": false, 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_bucket_count": 0, 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: "degraded_stretch_mode": 0, 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: "recovering_stretch_mode": 0, 2024-08-18T21:37:35.260 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_bucket": 0 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_metadata": [ 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6804/2216089247,v1:127.0.0.1:6805/2216089247]", 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:37:35.261 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:37:35.262 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:33:13.175443Z", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6802/2216089247,v1:127.0.0.1:6803/2216089247]", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:37:35.263 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2216089247,v1:127.0.0.1:6809/2216089247]", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2216089247,v1:127.0.0.1:6807/2216089247]", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/0", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:37:35.264 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6804/3809673106,v1:127.0.0.1:6805/3809673106]", 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:37:35.265 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:37:35.266 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:33:50.010420Z", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6802/3809673106,v1:127.0.0.1:6803/3809673106]", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3809673106,v1:127.0.0.1:6809/3809673106]", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3809673106,v1:127.0.0.1:6807/3809673106]", 2024-08-18T21:37:35.267 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/1", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.268 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/1808399366,v1:127.0.0.1:6813/1808399366]", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:37:35.269 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:34:28.689269Z", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:37:35.270 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/1808399366,v1:127.0.0.1:6811/1808399366]", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1808399366,v1:127.0.0.1:6817/1808399366]", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1808399366,v1:127.0.0.1:6815/1808399366]", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:37:35.271 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/2", 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_clean_epochs": { 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: "min_last_epoch_clean": 0, 2024-08-18T21:37:35.272 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": { 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: "per_pool": [ 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: "poolid": 1, 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: "floor": 33 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_epochs": [ 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 43 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.273 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:37:35.274 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 45 2024-08-18T21:37:35.274 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.274 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.274 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.274 INFO:tasks.workunit.client.0.smithi099.stderr:report 1645342976 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_first_committed": 1, 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_last_committed": 50, 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: "crushmap": { 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": [ 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.0", 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:37:35.275 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.1", 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.2", 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: "types": [ 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 0, 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd" 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.276 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "host" 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 2, 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "chassis" 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 3, 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "rack" 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 4, 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "row" 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.277 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 5, 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pdu" 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 6, 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pod" 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 7, 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "room" 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 8, 2024-08-18T21:37:35.278 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "datacenter" 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 9, 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "zone" 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 10, 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "region" 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "root" 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.279 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: "buckets": [ 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -1, 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default", 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:37:35.280 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -2, 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default~hdd", 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:37:35.281 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099", 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:37:35.282 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.283 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099~hdd", 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:37:35.284 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:37:35.285 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.285 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:37:35.285 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:37:35.285 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:37:35.285 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.285 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.285 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:37:35.285 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:37:35.285 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:37:35.285 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: "rules": [ 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_id": 0, 2024-08-18T21:37:35.286 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_name": "replicated_rule", 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: "steps": [ 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "take", 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: "item": -1, 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: "item_name": "default" 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "choose_firstn", 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 0, 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "osd" 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.287 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "emit" 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: "tunables": { 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_tries": 0, 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_fallback_tries": 0, 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_total_tries": 50, 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_descend_once": 1, 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_vary_r": 1, 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_stable": 1, 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_descents": 100, 2024-08-18T21:37:35.288 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_collision_tries": 100, 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "straw_calc_version": 1, 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "allowed_bucket_algs": 54, 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "profile": "jewel", 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_tunables": 1, 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "legacy_tunables": 0, 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "minimum_required_version": "jewel", 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables": 1, 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables2": 1, 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v2_rules": 0, 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables3": 1, 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v3_rules": 0, 2024-08-18T21:37:35.289 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v4_buckets": 1, 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables5": 1, 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v5_rules": 0, 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "has_msr_rules": 0 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_args": {} 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "fsmap": { 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "btime": "2024-08-18T21:33:06:194092+0000", 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "default_fscid": -1, 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": { 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": {}, 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "ro_compat": {}, 2024-08-18T21:37:35.290 INFO:tasks.workunit.client.0.smithi099.stdout: "incompat": { 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_1": "base v0.20", 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_2": "client writeable ranges", 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_3": "default file layouts on dirs", 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_4": "dir inode in separate object", 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_5": "mds uses versioned encoding", 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_8": "no anchor table", 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_9": "file layout v2", 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_10": "snaprealm v2", 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_11": "minor log segments", 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_12": "quiesce subvolumes" 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.291 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_flags": { 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: "enable_multiple": true, 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: "ever_enabled_multiple": true 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: "standbys": [], 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: "filesystems": [] 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_first_committed": 1, 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_last_committed": 1, 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: "auth": { 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 5, 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: "num_secrets": 3 2024-08-18T21:37:35.292 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 32, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_active": 32, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_unknown": 0, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osd": 3, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_sum": { 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 62050, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 100, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 0, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 300, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 100, 2024-08-18T21:37:35.293 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 100, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 94, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 94, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:37:35.294 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:37:35.295 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:37:35.296 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 101, 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 101, 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 64, 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 64, 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 0 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum": { 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 96, 2024-08-18T21:37:35.297 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83848, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2304, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314488952, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036686848, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:37:35.298 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2359296, 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 535132, 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10392, 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416936 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.299 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum_by_class": { 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "hdd": { 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 96, 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:37:35.300 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83848, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2304, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314488952, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036686848, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2359296, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 535132, 2024-08-18T21:37:35.301 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10392, 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416936 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:37:35.302 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_stats": [ 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.303 INFO:tasks.workunit.client.0.smithi099.stdout: "poolid": 1, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 32, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 62050, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 100, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 0, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 300, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 100, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 100, 2024-08-18T21:37:35.304 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 94, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 94, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:37:35.305 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:37:35.306 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 1220608, 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 185266, 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:37:35.307 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 101, 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 101, 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 64, 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 64, 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 3 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_stats": [ 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 21474836520 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:37:35.308 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 184683593749 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 193273528340 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_state": [ 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: "state": "active+undersized+degraded", 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 32 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.309 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_osd": [ 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 8, 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 32, 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 0, 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.310 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 24, 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 32, 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": [ 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": {} 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: "servicemap": { 2024-08-18T21:37:35.311 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 7, 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:37:17.917614+0000", 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: "services": {} 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_first_committed": 1, 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_last_committed": 123, 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_first_committed": 1, 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_last_committed": 494, 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: "paxos": { 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 669, 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pn": 0, 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: "accepted_pn": 0 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:37:35.312 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:37:35.313 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-18T21:37:35.313 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-08-18T21:37:35.313 INFO:tasks.workunit.client.0.smithi099.stdout:delay recovery 2024-08-18T21:37:35.314 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:37:35.671 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:37:35.672 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060700+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:49:40.518308+0000 0 0 2024-08-18T21:37:35.672 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.792738+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:10:03.086555+0000 0 0 2024-08-18T21:37:35.672 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.792672+0000 36'4 50:68 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:44:42.279447+0000 0 0 2024-08-18T21:37:35.672 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.794552+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:32:35.017116+0000 0 0 2024-08-18T21:37:35.672 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793126+0000 36'3 50:73 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:04:53.673735+0000 1 0 2024-08-18T21:37:35.672 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.793821+0000 36'5 50:75 [1,2] 1 [1,2] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:40:43.447167+0000 2 0 2024-08-18T21:37:35.672 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793178+0000 36'3 50:73 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:53:38.202575+0000 0 0 2024-08-18T21:37:35.672 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.796571+0000 36'2 50:72 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:17:19.819950+0000 1 0 2024-08-18T21:37:35.672 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.792482+0000 36'4 50:68 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:01:15.594403+0000 0 0 2024-08-18T21:37:35.673 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-18T21:36:19.060214+0000 36'6 50:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:27:39.408296+0000 0 0 2024-08-18T21:37:35.673 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.793062+0000 36'2 50:66 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-20T07:58:37.646145+0000 0 0 2024-08-18T21:37:35.673 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.678911+0000 36'5 50:75 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:24:52.944633+0000 0 0 2024-08-18T21:37:35.673 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.792497+0000 36'1 50:61 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:12:27.011984+0000 0 0 2024-08-18T21:37:35.673 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.796210+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:19:03.521863+0000 0 0 2024-08-18T21:37:35.673 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:19.060000+0000 36'2 50:59 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:38:20.186954+0000 0 0 2024-08-18T21:37:35.673 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.795623+0000 36'2 50:62 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:11:33.117016+0000 0 0 2024-08-18T21:37:35.673 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.792399+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:42:41.761298+0000 0 0 2024-08-18T21:37:35.673 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.059997+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:26:56.103631+0000 0 0 2024-08-18T21:37:35.673 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.792213+0000 50'5 50:75 [1,2] 1 [1,2] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:03:06.414446+0000 2 0 2024-08-18T21:37:35.674 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:19.060940+0000 36'4 50:61 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:16:08.923652+0000 0 0 2024-08-18T21:37:35.674 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060354+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:09:10.020078+0000 0 0 2024-08-18T21:37:35.674 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.795482+0000 36'3 50:63 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T08:44:39.434197+0000 0 0 2024-08-18T21:37:35.674 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.796728+0000 36'1 50:61 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:45:38.775344+0000 0 0 2024-08-18T21:37:35.674 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793454+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:16:56.012624+0000 0 0 2024-08-18T21:37:35.674 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.062198+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:06:12.002363+0000 0 0 2024-08-18T21:37:35.674 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.678901+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:56:17.057598+0000 0 0 2024-08-18T21:37:35.674 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.678909+0000 36'4 50:68 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:27:51.072853+0000 0 0 2024-08-18T21:37:35.675 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.794223+0000 36'1 50:61 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:55:35.538743+0000 0 0 2024-08-18T21:37:35.675 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060250+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:18:42.255563+0000 0 0 2024-08-18T21:37:35.675 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.794083+0000 36'2 50:66 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:16:51.117645+0000 0 0 2024-08-18T21:37:35.675 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.793596+0000 36'4 50:74 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:20:45.081929+0000 1 0 2024-08-18T21:37:35.675 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.795883+0000 36'5 50:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:31:12.592149+0000 0 0 2024-08-18T21:37:35.675 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:37:35.675 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:37:35.675 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:37:35.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-08-18T21:37:35.689 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.1 2024-08-18T21:37:35.689 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-18T21:37:35.689 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-18T21:37:35.690 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:37:35.690 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:37:35.690 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:37:35.690 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.1.asok 2024-08-18T21:37:35.691 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-08-18T21:37:35.691 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.1.asok set_recovery_delay 100000 2024-08-18T21:37:35.803 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:37:35.815 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-08-18T21:37:35.817 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-08-18T21:37:35.817 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-18T21:37:35.817 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-18T21:37:35.818 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:37:35.818 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:37:35.818 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:37:35.818 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.2.asok 2024-08-18T21:37:35.818 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-08-18T21:37:35.819 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.2.asok set_recovery_delay 100000 2024-08-18T21:37:35.928 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:37:35.939 INFO:tasks.workunit.client.0.smithi099.stdout:revive divergent 0 2024-08-18T21:37:35.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 0' 2024-08-18T21:37:35.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:37:36.360 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:37:36.360 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060700+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:49:40.518308+0000 0 0 2024-08-18T21:37:36.360 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.792738+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:10:03.086555+0000 0 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.792672+0000 36'4 50:68 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:44:42.279447+0000 0 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.794552+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:32:35.017116+0000 0 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793126+0000 36'3 50:73 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:04:53.673735+0000 1 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.793821+0000 36'5 50:75 [1,2] 1 [1,2] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:40:43.447167+0000 2 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793178+0000 36'3 50:73 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:53:38.202575+0000 0 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.796571+0000 36'2 50:72 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:17:19.819950+0000 1 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.792482+0000 36'4 50:68 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:01:15.594403+0000 0 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-18T21:36:19.060214+0000 36'6 50:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:27:39.408296+0000 0 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.793062+0000 36'2 50:66 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-20T07:58:37.646145+0000 0 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.678911+0000 36'5 50:75 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:24:52.944633+0000 0 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.792497+0000 36'1 50:61 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:12:27.011984+0000 0 0 2024-08-18T21:37:36.361 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.796210+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:19:03.521863+0000 0 0 2024-08-18T21:37:36.362 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:19.060000+0000 36'2 50:59 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:38:20.186954+0000 0 0 2024-08-18T21:37:36.362 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.795623+0000 36'2 50:62 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:11:33.117016+0000 0 0 2024-08-18T21:37:36.362 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.792399+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:42:41.761298+0000 0 0 2024-08-18T21:37:36.362 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.059997+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:26:56.103631+0000 0 0 2024-08-18T21:37:36.362 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.792213+0000 50'5 50:75 [1,2] 1 [1,2] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:03:06.414446+0000 2 0 2024-08-18T21:37:36.362 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:19.060940+0000 36'4 50:61 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:16:08.923652+0000 0 0 2024-08-18T21:37:36.362 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060354+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:09:10.020078+0000 0 0 2024-08-18T21:37:36.362 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.795482+0000 36'3 50:63 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T08:44:39.434197+0000 0 0 2024-08-18T21:37:36.362 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.796728+0000 36'1 50:61 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:45:38.775344+0000 0 0 2024-08-18T21:37:36.362 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793454+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:16:56.012624+0000 0 0 2024-08-18T21:37:36.362 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.062198+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:06:12.002363+0000 0 0 2024-08-18T21:37:36.363 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.678901+0000 36'3 50:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:56:17.057598+0000 0 0 2024-08-18T21:37:36.363 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.678909+0000 36'4 50:68 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:27:51.072853+0000 0 0 2024-08-18T21:37:36.363 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.794223+0000 36'1 50:61 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:55:35.538743+0000 0 0 2024-08-18T21:37:36.363 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060250+0000 36'3 50:60 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:18:42.255563+0000 0 0 2024-08-18T21:37:36.363 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.794083+0000 36'2 50:66 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:16:51.117645+0000 0 0 2024-08-18T21:37:36.363 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.793596+0000 36'4 50:74 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:20:45.081929+0000 1 0 2024-08-18T21:37:36.363 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.795883+0000 36'5 50:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:31:12.592149+0000 0 0 2024-08-18T21:37:36.363 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:37:36.363 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:37:36.364 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:37:36.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-08-18T21:37:37.029 INFO:tasks.workunit.client.0.smithi099.stderr:noup is set 2024-08-18T21:37:37.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-08-18T21:37:37.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:37:37.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:37:37.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-18T21:37:37.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:37:37.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-18T21:37:37.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:37:37.053 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:37:37.054 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:37:37.055 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:37:37.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:37:37.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:37:37.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:37:37.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:37:37.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:37:37.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:37:37.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:37:37.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:37:37.056 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:37:37.056 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:37:37.056 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-18T21:37:37.056 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:37:37.056 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-18T21:37:37.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:37:37.057 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-18T21:37:37.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-18T21:37:37.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:37:37.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:37:37.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:37:37.097 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:37:37.098+0000 7f8999d987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:37:37.113 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:37:37.114+0000 7f8999d987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:37:37.114 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:37:37.114+0000 7f8999d987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:37:37.487 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-08-18T21:37:37.487 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-08-18T21:37:38.399 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:37:38.398+0000 7f8999d987c0 -1 Falling back to public interface 2024-08-18T21:37:41.448 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:37:41.450+0000 7f8999d987c0 -1 osd.0 36 log_to_monitors true 2024-08-18T21:37:42.489 INFO:tasks.workunit.client.0.smithi099.stdout:delay recovery divergent 2024-08-18T21:37:42.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-08-18T21:37:42.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:37:42.918 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:37:42.918 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060700+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:49:40.518308+0000 0 0 2024-08-18T21:37:42.918 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.792738+0000 36'3 52:71 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:10:03.086555+0000 0 0 2024-08-18T21:37:42.918 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.792672+0000 36'4 52:72 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:44:42.279447+0000 0 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.794552+0000 36'3 52:71 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:32:35.017116+0000 0 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793126+0000 36'3 52:77 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:04:53.673735+0000 1 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.793821+0000 36'5 52:79 [1,2] 1 [1,2] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:40:43.447167+0000 2 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793178+0000 36'3 52:77 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:53:38.202575+0000 0 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.796571+0000 36'2 52:76 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:17:19.819950+0000 1 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.792482+0000 36'4 52:72 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:01:15.594403+0000 0 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-18T21:36:19.060214+0000 36'6 52:66 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:27:39.408296+0000 0 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.793062+0000 36'2 52:70 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-20T07:58:37.646145+0000 0 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.678911+0000 36'5 52:79 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:24:52.944633+0000 0 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.792497+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:12:27.011984+0000 0 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.796210+0000 36'3 52:71 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:19:03.521863+0000 0 0 2024-08-18T21:37:42.919 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:19.060000+0000 36'2 52:62 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:38:20.186954+0000 0 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.795623+0000 36'2 52:66 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:11:33.117016+0000 0 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.792399+0000 36'3 52:71 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:42:41.761298+0000 0 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.059997+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:26:56.103631+0000 0 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.792213+0000 50'5 52:79 [1,2] 1 [1,2] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:03:06.414446+0000 2 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:19.060940+0000 36'4 52:64 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:16:08.923652+0000 0 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060354+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:09:10.020078+0000 0 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.795482+0000 36'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T08:44:39.434197+0000 0 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.796728+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:45:38.775344+0000 0 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.793454+0000 36'3 52:71 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:16:56.012624+0000 0 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.062198+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:06:12.002363+0000 0 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.678901+0000 36'3 52:71 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:56:17.057598+0000 0 0 2024-08-18T21:37:42.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.678909+0000 36'4 52:72 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:27:51.072853+0000 0 0 2024-08-18T21:37:42.921 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.794223+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:55:35.538743+0000 0 0 2024-08-18T21:37:42.921 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060250+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:18:42.255563+0000 0 0 2024-08-18T21:37:42.921 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.794083+0000 36'2 52:70 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:16:51.117645+0000 0 0 2024-08-18T21:37:42.921 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:18.793596+0000 36'4 52:78 [1,2] 1 [1,2] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:20:45.081929+0000 1 0 2024-08-18T21:37:42.921 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.795883+0000 36'5 52:69 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:31:12.592149+0000 0 0 2024-08-18T21:37:42.921 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:37:42.921 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:37:42.921 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:37:42.934 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.0 2024-08-18T21:37:42.935 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-18T21:37:42.935 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-18T21:37:42.935 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:37:42.935 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:37:42.936 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:37:42.936 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.0.asok 2024-08-18T21:37:42.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-08-18T21:37:42.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.0.asok set_recovery_delay 100000 2024-08-18T21:37:43.043 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:37:43.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-08-18T21:37:43.793 INFO:tasks.workunit.client.0.smithi099.stderr:noup is unset 2024-08-18T21:37:43.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-08-18T21:37:43.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:37:43.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:37:43.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:37:43.815 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:37:43.815 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:37:43.815 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:37:43.815 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:37:43.816 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:37:43.816 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:37:44.366 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 56 up_thru 34 down_at 37 last_clean_interval [5,36) [v2:127.0.0.1:6818/2696221228,v1:127.0.0.1:6819/2696221228] [v2:127.0.0.1:6820/2696221228,v1:127.0.0.1:6821/2696221228] exists,up 6e2c559a-40e6-4a8a-8afe-cf8a54df7cbc 2024-08-18T21:37:44.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:37:44.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:37:44.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:37:44.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-08-18T21:37:44.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:37:44.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:37:44.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:37:44.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:37:44.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:37:44.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:37:44.368 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:37:44.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:37:44.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:37:44.883 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 43 up_thru 56 down_at 38 last_clean_interval [10,36) [v2:127.0.0.1:6802/3809673106,v1:127.0.0.1:6803/3809673106] [v2:127.0.0.1:6804/3809673106,v1:127.0.0.1:6805/3809673106] exists,up d2abeb81-0f38-419c-ba93-2eea17b6fc91 2024-08-18T21:37:44.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:37:44.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:37:44.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:37:44.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-08-18T21:37:44.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:37:44.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:37:44.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:37:44.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:37:44.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:37:44.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:37:44.885 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:37:44.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:37:44.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:37:45.481 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 45 up_thru 56 down_at 39 last_clean_interval [15,36) [v2:127.0.0.1:6810/1808399366,v1:127.0.0.1:6811/1808399366] [v2:127.0.0.1:6812/1808399366,v1:127.0.0.1:6813/1808399366] exists,up 33c0de0c-1a9c-4bb2-9e7b-3a49e559e1e4 2024-08-18T21:37:45.482 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:37:45.482 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:37:45.482 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:37:45.482 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:37:45.950 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:37:45.950 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060700+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:49:40.518308+0000 0 0 2024-08-18T21:37:45.950 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.168146+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:17:26.995548+0000 0 0 2024-08-18T21:37:45.950 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.112921+0000 36'4 57:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:54:32.875362+0000 0 0 2024-08-18T21:37:45.951 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.106968+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:22:29.638104+0000 0 0 2024-08-18T21:37:45.951 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.165017+0000 36'3 57:93 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:17:23.002134+0000 1 0 2024-08-18T21:37:45.951 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovering+degraded 2024-08-18T21:37:45.164638+0000 36'5 57:94 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 no scrub is scheduled 2 0 2024-08-18T21:37:45.951 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.166262+0000 36'3 57:93 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:12:54.134304+0000 0 0 2024-08-18T21:37:45.951 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.111628+0000 36'2 57:92 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:17:12.462169+0000 1 0 2024-08-18T21:37:45.951 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.165948+0000 36'4 57:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:05:39.233133+0000 0 0 2024-08-18T21:37:45.952 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-18T21:36:19.060214+0000 36'6 52:66 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:27:39.408296+0000 0 0 2024-08-18T21:37:45.952 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.112569+0000 36'2 57:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-19T22:14:22.471756+0000 0 0 2024-08-18T21:37:45.952 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:37:45.167488+0000 36'5 57:95 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-19T22:59:05.939906+0000 0 0 2024-08-18T21:37:45.952 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.792497+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:12:27.011984+0000 0 0 2024-08-18T21:37:45.952 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.111927+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:14:49.363222+0000 0 0 2024-08-18T21:37:45.952 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:19.060000+0000 36'2 52:62 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:38:20.186954+0000 0 0 2024-08-18T21:37:45.953 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.795623+0000 36'2 52:66 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:11:33.117016+0000 0 0 2024-08-18T21:37:45.953 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.169157+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:52:22.012057+0000 0 0 2024-08-18T21:37:45.953 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.059997+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:26:56.103631+0000 0 0 2024-08-18T21:37:45.953 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovery_wait+degraded 2024-08-18T21:37:45.113159+0000 50'5 57:93 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 no scrub is scheduled 2 0 2024-08-18T21:37:45.953 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:19.060940+0000 36'4 52:64 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:16:08.923652+0000 0 0 2024-08-18T21:37:45.954 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060354+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:09:10.020078+0000 0 0 2024-08-18T21:37:45.954 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.795482+0000 36'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T08:44:39.434197+0000 0 0 2024-08-18T21:37:45.954 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.796728+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:45:38.775344+0000 0 0 2024-08-18T21:37:45.954 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.110235+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:39:46.945196+0000 0 0 2024-08-18T21:37:45.954 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.062198+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:06:12.002363+0000 0 0 2024-08-18T21:37:45.954 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.167245+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:20:53.803907+0000 0 0 2024-08-18T21:37:45.955 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.112235+0000 36'4 57:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:53:58.669077+0000 0 0 2024-08-18T21:37:45.955 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.794223+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:55:35.538743+0000 0 0 2024-08-18T21:37:45.955 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060250+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:18:42.255563+0000 0 0 2024-08-18T21:37:45.955 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.164232+0000 36'2 57:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:19:20.776890+0000 0 0 2024-08-18T21:37:45.955 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.165036+0000 36'4 57:94 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:24:02.944156+0000 1 0 2024-08-18T21:37:45.955 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.795883+0000 36'5 52:69 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:31:12.592149+0000 0 0 2024-08-18T21:37:45.956 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:37:45.956 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:37:45.956 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:37:45.967 INFO:tasks.workunit.client.0.smithi099.stdout:wait for peering 2024-08-18T21:37:45.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-08-18T21:37:45.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:37:46.397 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:37:46.397 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060700+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:49:40.518308+0000 0 0 2024-08-18T21:37:46.397 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.168146+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:17:26.995548+0000 0 0 2024-08-18T21:37:46.398 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.112921+0000 36'4 57:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:54:32.875362+0000 0 0 2024-08-18T21:37:46.398 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.106968+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:22:29.638104+0000 0 0 2024-08-18T21:37:46.398 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.165017+0000 36'3 57:93 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:17:23.002134+0000 1 0 2024-08-18T21:37:46.398 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovering+degraded 2024-08-18T21:37:45.164638+0000 36'5 57:94 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 no scrub is scheduled 2 0 2024-08-18T21:37:46.398 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.166262+0000 36'3 57:93 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:12:54.134304+0000 0 0 2024-08-18T21:37:46.398 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.111628+0000 36'2 57:92 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:17:12.462169+0000 1 0 2024-08-18T21:37:46.398 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.165948+0000 36'4 57:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:05:39.233133+0000 0 0 2024-08-18T21:37:46.398 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-18T21:36:19.060214+0000 36'6 52:66 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:27:39.408296+0000 0 0 2024-08-18T21:37:46.398 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.112569+0000 36'2 57:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-19T22:14:22.471756+0000 0 0 2024-08-18T21:37:46.398 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:37:45.167488+0000 36'5 57:95 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-19T22:59:05.939906+0000 0 0 2024-08-18T21:37:46.398 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.792497+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:12:27.011984+0000 0 0 2024-08-18T21:37:46.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.111927+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:14:49.363222+0000 0 0 2024-08-18T21:37:46.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:19.060000+0000 36'2 52:62 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:38:20.186954+0000 0 0 2024-08-18T21:37:46.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.795623+0000 36'2 52:66 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:11:33.117016+0000 0 0 2024-08-18T21:37:46.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.169157+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:52:22.012057+0000 0 0 2024-08-18T21:37:46.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.059997+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:26:56.103631+0000 0 0 2024-08-18T21:37:46.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovery_wait+degraded 2024-08-18T21:37:45.113159+0000 50'5 57:93 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 no scrub is scheduled 2 0 2024-08-18T21:37:46.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:19.060940+0000 36'4 52:64 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:16:08.923652+0000 0 0 2024-08-18T21:37:46.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060354+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:09:10.020078+0000 0 0 2024-08-18T21:37:46.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.795482+0000 36'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T08:44:39.434197+0000 0 0 2024-08-18T21:37:46.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.796728+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:45:38.775344+0000 0 0 2024-08-18T21:37:46.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.110235+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:39:46.945196+0000 0 0 2024-08-18T21:37:46.400 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.062198+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:06:12.002363+0000 0 0 2024-08-18T21:37:46.400 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.167245+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:20:53.803907+0000 0 0 2024-08-18T21:37:46.400 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.112235+0000 36'4 57:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:53:58.669077+0000 0 0 2024-08-18T21:37:46.400 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.794223+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:55:35.538743+0000 0 0 2024-08-18T21:37:46.400 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060250+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:18:42.255563+0000 0 0 2024-08-18T21:37:46.400 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.164232+0000 36'2 57:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:19:20.776890+0000 0 0 2024-08-18T21:37:46.400 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.165036+0000 36'4 57:94 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:24:02.944156+0000 1 0 2024-08-18T21:37:46.400 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.795883+0000 36'5 52:69 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:31:12.592149+0000 0 0 2024-08-18T21:37:46.400 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:37:46.400 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:37:46.400 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:37:46.413 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-08-18T21:37:46.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 0' 2024-08-18T21:37:46.541 INFO:tasks.workunit.client.0.smithi099.stdout:killing divergent 0 2024-08-18T21:37:46.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:37:46.944 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:37:46.944 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060700+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:49:40.518308+0000 0 0 2024-08-18T21:37:46.944 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.168146+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:17:26.995548+0000 0 0 2024-08-18T21:37:46.944 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.112921+0000 36'4 57:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:54:32.875362+0000 0 0 2024-08-18T21:37:46.944 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.106968+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:22:29.638104+0000 0 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.165017+0000 36'3 57:93 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:17:23.002134+0000 1 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovering+degraded 2024-08-18T21:37:45.164638+0000 36'5 57:94 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 no scrub is scheduled 2 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.166262+0000 36'3 57:93 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:12:54.134304+0000 0 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.111628+0000 36'2 57:92 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:17:12.462169+0000 1 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.165948+0000 36'4 57:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:05:39.233133+0000 0 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-08-18T21:36:19.060214+0000 36'6 52:66 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:27:39.408296+0000 0 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.112569+0000 36'2 57:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-19T22:14:22.471756+0000 0 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:37:45.167488+0000 36'5 57:95 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-19T22:59:05.939906+0000 0 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.792497+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:12:27.011984+0000 0 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.111927+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:14:49.363222+0000 0 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:19.060000+0000 36'2 52:62 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:38:20.186954+0000 0 0 2024-08-18T21:37:46.945 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-08-18T21:36:18.795623+0000 36'2 52:66 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:11:33.117016+0000 0 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.169157+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:52:22.012057+0000 0 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.059997+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:26:56.103631+0000 0 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovery_wait+degraded 2024-08-18T21:37:45.113159+0000 50'5 57:93 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 no scrub is scheduled 2 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-08-18T21:36:19.060940+0000 36'4 52:64 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:16:08.923652+0000 0 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060354+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:09:10.020078+0000 0 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:18.795482+0000 36'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T08:44:39.434197+0000 0 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.796728+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:45:38.775344+0000 0 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.110235+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:39:46.945196+0000 0 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.062198+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:06:12.002363+0000 0 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.167245+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:20:53.803907+0000 0 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.112235+0000 36'4 57:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:53:58.669077+0000 0 0 2024-08-18T21:37:46.946 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-08-18T21:36:18.794223+0000 36'1 52:65 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:55:35.538743+0000 0 0 2024-08-18T21:37:46.947 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-08-18T21:36:19.060250+0000 36'3 52:63 [2,1] 2 [2,1] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:18:42.255563+0000 0 0 2024-08-18T21:37:46.947 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.164232+0000 36'2 57:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:19:20.776890+0000 0 0 2024-08-18T21:37:46.947 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.165036+0000 36'4 57:94 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:24:02.944156+0000 1 0 2024-08-18T21:37:46.947 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-08-18T21:36:18.795883+0000 36'5 52:69 [1,2] 1 [1,2] 1 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:31:12.592149+0000 0 0 2024-08-18T21:37:46.947 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:37:46.947 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:37:46.947 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:37:46.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.0 2024-08-18T21:37:46.961 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:37:46.961 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:37:46.961 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:37:46.961 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:37:46.961 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:37:47.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:37:47.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.27423.out 2024-08-18T21:37:47.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 0 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27423.out 2024-08-18T21:37:47.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-18T21:37:47.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-18T21:37:47.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-18T21:37:47.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-18T21:37:47.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-08-18T21:37:47.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.27423.out 2024-08-18T21:37:48.752 INFO:tasks.workunit.client.0.smithi099.stderr:Exporting 1.1b info 1.1b( v 36'5 lc 0'0 (0'0,36'5] local-lis/les=56/57 n=5 ec=32/23 lis/c=56/34 les/c/f=57/36/0 sis=56) 2024-08-18T21:37:48.753 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-08-18T21:37:48.759 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-08-18T21:37:48.760 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:dd677bd3:::existing_35:head# 2024-08-18T21:37:48.767 INFO:tasks.workunit.client.0.smithi099.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-08-18T21:37:48.768 INFO:tasks.workunit.client.0.smithi099.stderr:Export successful 2024-08-18T21:37:48.768 INFO:tasks.workunit.client.0.smithi099.stdout: marking collection for removal 2024-08-18T21:37:48.769 INFO:tasks.workunit.client.0.smithi099.stdout:setting '_remove' omap key 2024-08-18T21:37:48.769 INFO:tasks.workunit.client.0.smithi099.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-08-18T21:37:48.868 INFO:tasks.workunit.client.0.smithi099.stderr:Remove successful 2024-08-18T21:37:49.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 0 --op import --file td/divergent-priors/exp.27423.out 2024-08-18T21:37:49.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-18T21:37:49.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-18T21:37:49.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-18T21:37:49.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-18T21:37:49.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-08-18T21:37:49.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op import --file td/divergent-priors/exp.27423.out 2024-08-18T21:37:50.735 INFO:tasks.workunit.client.0.smithi099.stdout:get_pg_num_history pg_num_history pg_num_history(e57 pg_nums {1={23=1,32=32}} deleted_pools ) 2024-08-18T21:37:50.735 INFO:tasks.workunit.client.0.smithi099.stdout:Importing pgid 1.1b 2024-08-18T21:37:50.735 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-08-18T21:37:50.735 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:37:50.831 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-08-18T21:37:50.831 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:37:50.873 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:dd677bd3:::existing_35:head# 2024-08-18T21:37:50.873 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:37:50.914 INFO:tasks.workunit.client.0.smithi099.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-08-18T21:37:50.915 INFO:tasks.workunit.client.0.smithi099.stdout:snapset 0=[]:{} 2024-08-18T21:37:50.956 INFO:tasks.workunit.client.0.smithi099.stdout:write_pg epoch 57 info 1.1b( v 36'5 lc 0'0 (0'0,36'5] local-lis/les=56/57 n=5 ec=32/23 lis/c=56/34 les/c/f=57/36/0 sis=56) 2024-08-18T21:37:50.998 INFO:tasks.workunit.client.0.smithi099.stdout:Import successful 2024-08-18T21:37:51.538 INFO:tasks.workunit.client.0.smithi099.stdout:reviving divergent 0 2024-08-18T21:37:51.538 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 0' 2024-08-18T21:37:51.538 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:37:51.957 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:37:51.957 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.107774+0000 36'3 57:64 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:21:32.371159+0000 0 0 2024-08-18T21:37:51.957 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.168146+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:17:26.995548+0000 0 0 2024-08-18T21:37:51.957 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.112921+0000 36'4 57:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:54:32.875362+0000 0 0 2024-08-18T21:37:51.957 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.106968+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:22:29.638104+0000 0 0 2024-08-18T21:37:51.957 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.165017+0000 36'3 57:93 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:17:23.002134+0000 1 0 2024-08-18T21:37:51.957 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovering+degraded 2024-08-18T21:37:45.164638+0000 36'5 57:94 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 no scrub is scheduled 2 0 2024-08-18T21:37:51.957 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.166262+0000 36'3 57:93 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:12:54.134304+0000 0 0 2024-08-18T21:37:51.957 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.111628+0000 36'2 57:92 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:17:12.462169+0000 1 0 2024-08-18T21:37:51.957 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:37:45.165948+0000 57'5 57:89 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:05:39.233133+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-08-18T21:37:45.107640+0000 36'6 57:82 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:23:36.442546+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.112569+0000 36'2 57:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-19T22:14:22.471756+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:37:45.167488+0000 36'5 57:95 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-19T22:59:05.939906+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:37:45.111321+0000 36'1 57:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T00:36:07.490983+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.111927+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:14:49.363222+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.109970+0000 36'2 57:78 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T21:43:11.080461+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.109051+0000 36'2 57:63 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T03:18:32.498174+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.169157+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:52:22.012057+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.113004+0000 36'3 57:79 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:54:20.006441+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovery_wait+degraded 2024-08-18T21:37:45.113159+0000 50'5 57:93 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 no scrub is scheduled 2 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.113929+0000 36'4 57:80 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:48:18.318308+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.107519+0000 36'3 57:64 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:08:13.348866+0000 0 0 2024-08-18T21:37:51.958 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 3 1 1 0 0 1851 0 0 3 0 3 active+recovery_wait+degraded 2024-08-18T21:37:45.110843+0000 36'3 57:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 no scrub is scheduled 0 0 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:37:45.108647+0000 36'1 57:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:02:51.974836+0000 0 0 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.110235+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:39:46.945196+0000 0 0 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.114883+0000 36'3 57:79 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T04:50:12.150458+0000 0 0 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.167245+0000 36'3 57:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:20:53.803907+0000 0 0 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.112235+0000 36'4 57:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:53:58.669077+0000 0 0 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:37:45.109438+0000 36'1 57:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:02:16.161907+0000 0 0 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:45.108781+0000 36'3 57:79 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:08:04.247536+0000 0 0 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:45.164232+0000 36'2 57:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T06:19:20.776890+0000 0 0 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:45.165036+0000 36'4 57:94 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T03:24:02.944156+0000 1 0 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-08-18T21:37:45.110351+0000 36'5 57:64 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 no scrub is scheduled 0 0 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:37:51.959 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:37:51.960 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:37:51.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-08-18T21:37:51.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:37:51.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:37:51.975 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:37:51.975 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:37:51.975 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:37:51.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:37:51.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:37:51.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:37:51.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-18T21:37:51.977 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:37:51.978 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-18T21:37:51.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:37:51.979 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-18T21:37:51.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-18T21:37:51.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:37:51.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:37:51.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:37:52.016 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:37:52.018+0000 7ff8ecf157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:37:52.018 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:37:52.018+0000 7ff8ecf157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:37:52.019 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:37:52.018+0000 7ff8ecf157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:37:52.450 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-18T21:37:52.450 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:37:52.450 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:37:52.450 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:37:52.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:37:52.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:37:52.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:37:52.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:37:52.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:37:52.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:37:52.887 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 56 up_thru 56 down_at 37 last_clean_interval [5,36) [v2:127.0.0.1:6818/2696221228,v1:127.0.0.1:6819/2696221228] [v2:127.0.0.1:6820/2696221228,v1:127.0.0.1:6821/2696221228] exists,up 6e2c559a-40e6-4a8a-8afe-cf8a54df7cbc 2024-08-18T21:37:52.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:37:52.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:37:52.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:37:52.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 0 2024-08-18T21:37:52.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:37:52.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:37:52.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:37:52.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:37:52.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:37:52.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:37:52.889 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:37:52.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:37:52.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:37:53.323 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:37:53.322+0000 7ff8ecf157c0 -1 Falling back to public interface 2024-08-18T21:37:53.341 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 56 up_thru 56 down_at 37 last_clean_interval [5,36) [v2:127.0.0.1:6818/2696221228,v1:127.0.0.1:6819/2696221228] [v2:127.0.0.1:6820/2696221228,v1:127.0.0.1:6821/2696221228] exists,up 6e2c559a-40e6-4a8a-8afe-cf8a54df7cbc 2024-08-18T21:37:53.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:37:53.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:37:53.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:37:53.342 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-08-18T21:37:56.248 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:37:56.250+0000 7ff8ecf157c0 -1 osd.0 57 log_to_monitors true 2024-08-18T21:38:13.345 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.0 2024-08-18T21:38:13.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-18T21:38:13.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-18T21:38:13.346 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:38:13.346 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:38:13.346 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:38:13.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.0.asok 2024-08-18T21:38:13.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-08-18T21:38:13.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.0.asok dump_ops_in_flight 2024-08-18T21:38:13.462 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:38:13.462 INFO:tasks.workunit.client.0.smithi099.stdout: "ops": [], 2024-08-18T21:38:13.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_ops": 0 2024-08-18T21:38:13.462 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:38:13.474 INFO:tasks.workunit.client.0.smithi099.stdout:allowing recovery 2024-08-18T21:38:13.474 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-08-18T21:38:13.474 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:38:13.899 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:38:13.899 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.266379+0000 36'3 63:78 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:40:25.701784+0000 0 0 2024-08-18T21:38:13.899 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.274308+0000 36'3 63:104 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:40:36.388940+0000 0 0 2024-08-18T21:38:13.899 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:58.273796+0000 36'4 63:104 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:00:01.277511+0000 0 0 2024-08-18T21:38:13.899 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.276411+0000 36'3 63:103 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:02:29.965845+0000 0 0 2024-08-18T21:38:13.899 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.275229+0000 36'3 63:109 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:20:53.099354+0000 1 0 2024-08-18T21:38:13.900 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-08-18T21:37:58.276756+0000 36'5 63:108 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 no scrub is scheduled 2 0 2024-08-18T21:38:13.900 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.263942+0000 36'3 63:109 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:44:51.822802+0000 0 0 2024-08-18T21:38:13.900 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:58.269721+0000 36'2 63:108 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:20:59.237877+0000 1 0 2024-08-18T21:38:13.900 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:37:58.274674+0000 57'5 63:105 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T01:01:56.154267+0000 0 0 2024-08-18T21:38:13.900 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-08-18T21:37:58.924104+0000 36'6 63:98 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:01:02.242263+0000 0 0 2024-08-18T21:38:13.900 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:58.275055+0000 36'2 63:102 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-19T21:54:11.968564+0000 0 0 2024-08-18T21:38:13.900 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:37:58.262900+0000 36'5 63:111 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:31:03.906589+0000 0 0 2024-08-18T21:38:13.900 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:37:58.265680+0000 36'1 63:76 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:21:16.359884+0000 0 0 2024-08-18T21:38:13.900 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.268234+0000 36'3 63:103 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:58:00.084730+0000 0 0 2024-08-18T21:38:13.900 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:58.802110+0000 36'2 63:94 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:57:12.942504+0000 0 0 2024-08-18T21:38:13.900 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:58.275469+0000 36'2 63:77 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:44:11.462333+0000 0 0 2024-08-18T21:38:13.901 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.273454+0000 36'3 63:104 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T01:50:46.186363+0000 0 0 2024-08-18T21:38:13.901 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.921069+0000 36'3 63:95 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:25:47.203991+0000 0 0 2024-08-18T21:38:13.901 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovering+degraded 2024-08-18T21:37:58.428729+0000 50'5 63:108 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 no scrub is scheduled 2 0 2024-08-18T21:38:13.901 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:58.923469+0000 36'4 63:96 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:22:56.447027+0000 0 0 2024-08-18T21:38:13.901 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.267143+0000 36'3 63:78 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:28:24.408047+0000 0 0 2024-08-18T21:38:13.901 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.428190+0000 36'3 63:81 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T05:52:43.545909+0000 0 0 2024-08-18T21:38:13.901 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:37:58.276531+0000 36'1 63:76 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:15:49.985349+0000 0 0 2024-08-18T21:38:13.901 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.276170+0000 36'3 63:103 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:58:34.336546+0000 0 0 2024-08-18T21:38:13.901 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.922876+0000 36'3 63:95 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:10:24.102379+0000 0 0 2024-08-18T21:38:13.901 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.269434+0000 36'3 63:103 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:47:46.908869+0000 0 0 2024-08-18T21:38:13.901 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:58.213138+0000 36'4 63:104 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:14:21.209861+0000 0 0 2024-08-18T21:38:13.902 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:37:58.274533+0000 36'1 63:76 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:17:59.921496+0000 0 0 2024-08-18T21:38:13.902 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.802113+0000 36'3 63:95 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:29:20.935834+0000 0 0 2024-08-18T21:38:13.902 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:58.276728+0000 36'2 63:102 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:28:20.031653+0000 0 0 2024-08-18T21:38:13.902 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:58.275478+0000 36'4 63:110 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T04:49:03.261435+0000 1 0 2024-08-18T21:38:13.902 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-08-18T21:37:58.267631+0000 36'5 63:76 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 no scrub is scheduled 0 0 2024-08-18T21:38:13.902 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:38:13.902 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:38:13.902 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:38:13.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-08-18T21:38:13.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-08-18T21:38:14.070 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:38:14.085 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-08-18T21:38:14.085 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-08-18T21:38:14.238 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:38:14.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-08-18T21:38:14.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-08-18T21:38:14.503 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:38:14.518 INFO:tasks.workunit.client.0.smithi099.stdout:reading divergent objects 2024-08-18T21:38:14.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-08-18T21:38:14.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-08-18T21:38:14.938 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:38:14.938 INFO:tasks.workunit.client.0.smithi099.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.266379+0000 36'3 63:78 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.992285+0000 0'0 2024-08-18T21:35:24.992285+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:40:25.701784+0000 0 0 2024-08-18T21:38:14.938 INFO:tasks.workunit.client.0.smithi099.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.274308+0000 36'3 63:104 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:18.524252+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T05:40:36.388940+0000 0 0 2024-08-18T21:38:14.938 INFO:tasks.workunit.client.0.smithi099.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:58.273796+0000 36'4 63:104 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:21.531841+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:00:01.277511+0000 0 0 2024-08-18T21:38:14.938 INFO:tasks.workunit.client.0.smithi099.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.276411+0000 36'3 63:103 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:25.594634+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:02:29.965845+0000 0 0 2024-08-18T21:38:14.938 INFO:tasks.workunit.client.0.smithi099.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.275229+0000 36'3 63:109 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:29.535524+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:20:53.099354+0000 1 0 2024-08-18T21:38:14.938 INFO:tasks.workunit.client.0.smithi099.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-08-18T21:37:58.276756+0000 36'5 63:108 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:28.529277+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 no scrub is scheduled 2 0 2024-08-18T21:38:14.938 INFO:tasks.workunit.client.0.smithi099.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.263942+0000 36'3 63:109 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:33.539154+0000 0'0 2024-08-18T21:35:33.539154+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:44:51.822802+0000 0 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:58.269721+0000 36'2 63:108 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:31.654768+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:20:59.237877+0000 1 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.6 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:37:58.274674+0000 57'5 63:105 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:27.560946+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T01:01:56.154267+0000 0 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-08-18T21:37:58.924104+0000 36'6 63:98 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:24.402461+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:01:02.242263+0000 0 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:58.275055+0000 36'2 63:102 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:24.577214+0000 0'0 2024-08-18T21:35:24.577214+0000 0 0 periodic scrub scheduled @ 2024-08-19T21:54:11.968564+0000 0 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-08-18T21:37:58.262900+0000 36'5 63:111 [1,2,0] 1 [1,2,0] 1 0'0 2024-08-18T21:35:16.488668+0000 0'0 2024-08-18T21:35:16.488668+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:31:03.906589+0000 0 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:37:58.265680+0000 36'1 63:76 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:21.097003+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T02:21:16.359884+0000 0 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.268234+0000 36'3 63:103 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:22.539726+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:58:00.084730+0000 0 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:58.802110+0000 36'2 63:94 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:20.431112+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:57:12.942504+0000 0 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:58.275469+0000 36'2 63:77 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:26.929415+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:44:11.462333+0000 0 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.273454+0000 36'3 63:104 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:15.448911+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T01:50:46.186363+0000 0 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.921069+0000 36'3 63:95 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:21.383132+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:25:47.203991+0000 0 0 2024-08-18T21:38:14.939 INFO:tasks.workunit.client.0.smithi099.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovering+degraded 2024-08-18T21:37:58.428729+0000 50'5 63:108 [1,2,0] 1 [1,2,0] 1 36'2 2024-08-18T21:35:32.549780+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 no scrub is scheduled 2 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:58.923469+0000 36'4 63:96 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:23.406151+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:22:56.447027+0000 0 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.267143+0000 36'3 63:78 [0,2,1] 0 [0,2,1] 0 0'0 2024-08-18T21:35:24.038248+0000 0'0 2024-08-18T21:35:24.038248+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:28:24.408047+0000 0 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.11 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.428190+0000 36'3 63:81 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:22.093253+0000 0'0 2024-08-18T21:35:03.313989+0000 0 0 periodic scrub scheduled @ 2024-08-20T05:52:43.545909+0000 0 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:37:58.276531+0000 36'1 63:76 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:23.047930+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T23:15:49.985349+0000 0 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.276170+0000 36'3 63:103 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:23.572060+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T08:58:34.336546+0000 0 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.922876+0000 36'3 63:95 [2,0,1] 2 [2,0,1] 2 0'0 2024-08-18T21:35:25.419119+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T02:10:24.102379+0000 0 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.269434+0000 36'3 63:103 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:26.643720+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T07:47:46.908869+0000 0 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:58.213138+0000 36'4 63:104 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:19.516381+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:14:21.209861+0000 0 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-08-18T21:37:58.274533+0000 36'1 63:76 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:20.121160+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T00:17:59.921496+0000 0 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-08-18T21:37:58.802113+0000 36'3 63:95 [2,1,0] 2 [2,1,0] 2 0'0 2024-08-18T21:35:22.400849+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T01:29:20.935834+0000 0 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-08-18T21:37:58.276728+0000 36'2 63:102 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:35:20.571388+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-19T22:28:20.031653+0000 0 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-08-18T21:37:58.275478+0000 36'4 63:110 [1,2,0] 1 [1,2,0] 1 36'1 2024-08-18T21:35:30.624645+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 periodic scrub scheduled @ 2024-08-20T04:49:03.261435+0000 1 0 2024-08-18T21:38:14.940 INFO:tasks.workunit.client.0.smithi099.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-08-18T21:37:58.267631+0000 36'5 63:76 [0,1,2] 0 [0,1,2] 0 0'0 2024-08-18T21:35:25.958838+0000 0'0 2024-08-18T21:35:03.313989+0000 0 1 no scrub is scheduled 0 0 2024-08-18T21:38:14.941 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:38:14.941 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:38:14.941 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:38:14.954 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-08-18T21:38:14.956 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-08-18T21:38:14.957 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:38:14.957 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-08-18T21:38:14.991 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:38:14.991 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-08-18T21:38:15.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:38:15.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-08-18T21:38:15.058 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:38:15.058 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-08-18T21:38:15.093 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-08-18T21:38:15.094 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-18T21:38:15.094 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-08-18T21:38:15.128 INFO:tasks.workunit.client.0.smithi099.stderr:error getting test/newobject_1: (2) No such file or directory 2024-08-18T21:38:15.129 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-08-18T21:38:15.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-08-18T21:38:15.164 INFO:tasks.workunit.client.0.smithi099.stderr:error getting test/newobject_2: (2) No such file or directory 2024-08-18T21:38:15.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-08-18T21:38:15.166 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-08-18T21:38:15.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-18T21:38:15.183 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 36'6 (36'1) modify 1:97db2a0f:::existing_2:head by client.4660.0:1 2024-08-18T21:35:51.665690+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:38:15.183 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 20 _merge_object_divergent_entries: keeping 36'6 (36'1) modify 1:97db2a0f:::existing_2:head by client.4660.0:1 2024-08-18T21:35:51.665690+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:38:15.183 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:38:15.183 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 36'1 first_divergent_update: 36'6 last_divergent_update: 36'6 2024-08-18T21:38:15.183 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-08-18T21:38:15.183 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-08-18T21:38:15.183 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 36'6 2024-08-18T21:38:15.183 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 36'6 (36'1) modify 1:97db2a0f:::existing_2:head by client.4660.0:1 2024-08-18T21:35:51.665690+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:38:15.183 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-08-18T21:38:15.184 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 36'6 (36'1) delete 1:70b3bf12:::existing_4:head by client.4667.0:1 2024-08-18T21:35:51.679497+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:38:15.184 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 20 _merge_object_divergent_entries: keeping 36'6 (36'1) delete 1:70b3bf12:::existing_4:head by client.4667.0:1 2024-08-18T21:35:51.679497+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:38:15.184 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-08-18T21:38:15.184 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 36'1 first_divergent_update: 36'6 last_divergent_update: 36'6 2024-08-18T21:38:15.184 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head has no more recent entries in log 2024-08-18T21:38:15.184 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head must be rolled back or recovered, attempting to rollback 2024-08-18T21:38:15.184 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head olog_can_rollback_to: 36'6 2024-08-18T21:38:15.184 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head cannot rollback 36'6 (36'1) delete 1:70b3bf12:::existing_4:head by client.4667.0:1 2024-08-18T21:35:51.679497+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:38:15.190 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head cannot roll back, removing and adding to missing 2024-08-18T21:38:15.190 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 36'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4662.0:1 2024-08-18T21:35:51.671644+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:38:15.190 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 20 _merge_object_divergent_entries: keeping 36'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4662.0:1 2024-08-18T21:35:51.671644+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:38:15.190 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:38:15.190 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 36'5 last_divergent_update: 36'5 2024-08-18T21:38:15.190 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-08-18T21:38:15.190 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:37:44.398+0000 7fcc29a1c640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-08-18T21:38:15.190 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 36'6 (36'1) modify 1:d858c035:::existing_1:head by client.4657.0:1 2024-08-18T21:35:51.662870+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:38:15.190 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 20 _merge_object_divergent_entries: keeping 36'6 (36'1) modify 1:d858c035:::existing_1:head by client.4657.0:1 2024-08-18T21:35:51.662870+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:38:15.190 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:38:15.191 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 36'1 first_divergent_update: 36'6 last_divergent_update: 36'6 2024-08-18T21:38:15.191 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-08-18T21:38:15.191 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-08-18T21:38:15.191 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 36'7 2024-08-18T21:38:15.191 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 36'6 (36'1) modify 1:d858c035:::existing_1:head by client.4657.0:1 2024-08-18T21:35:51.662870+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:38:15.191 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-08-18T21:38:15.191 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 36'7 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4669.0:1 2024-08-18T21:35:51.681759+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:38:15.191 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 20 _merge_object_divergent_entries: keeping 36'7 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4669.0:1 2024-08-18T21:35:51.681759+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:38:15.191 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 36'7 last_divergent_update: 36'7 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.398+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.542+0000 7f897cd20640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 36'4 (36'1) delete 1:8881e6d8:::existing_3:head by client.4671.0:1 2024-08-18T21:35:51.684632+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.542+0000 7f897cd20640 20 _merge_object_divergent_entries: keeping 36'4 (36'1) delete 1:8881e6d8:::existing_3:head by client.4671.0:1 2024-08-18T21:35:51.684632+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.542+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.542+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 36'1 first_divergent_update: 36'4 last_divergent_update: 36'4 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.542+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.542+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.542+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 36'4 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.542+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 36'4 (36'1) delete 1:8881e6d8:::existing_3:head by client.4671.0:1 2024-08-18T21:35:51.684632+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.542+0000 7f897cd20640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-08-18T21:38:15.192 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 36'6 (36'1) modify 1:97db2a0f:::existing_2:head by client.4660.0:1 2024-08-18T21:35:51.665690+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:38:15.193 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 20 _merge_object_divergent_entries: keeping 36'6 (36'1) modify 1:97db2a0f:::existing_2:head by client.4660.0:1 2024-08-18T21:35:51.665690+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:38:15.193 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:38:15.193 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 36'1 first_divergent_update: 36'6 last_divergent_update: 36'6 2024-08-18T21:38:15.193 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-08-18T21:38:15.193 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-08-18T21:38:15.193 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 36'6 2024-08-18T21:38:15.193 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 36'6 (36'1) modify 1:97db2a0f:::existing_2:head by client.4660.0:1 2024-08-18T21:35:51.665690+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:38:15.194 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-08-18T21:38:15.194 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 36'6 (36'1) delete 1:70b3bf12:::existing_4:head by client.4667.0:1 2024-08-18T21:35:51.679497+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:38:15.194 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 20 _merge_object_divergent_entries: keeping 36'6 (36'1) delete 1:70b3bf12:::existing_4:head by client.4667.0:1 2024-08-18T21:35:51.679497+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-08-18T21:38:15.194 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:38:15.194 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 36'1 first_divergent_update: 36'6 last_divergent_update: 36'6 2024-08-18T21:38:15.194 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: more recent entry found: 50'5 (36'1) modify 1:70b3bf12:::existing_4:head by client.4788.0:1 2024-08-18T21:36:56.858016+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-08-18T21:38:15.194 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 36'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4662.0:1 2024-08-18T21:35:51.671644+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:38:15.194 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 20 _merge_object_divergent_entries: keeping 36'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4662.0:1 2024-08-18T21:35:51.671644+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:38:15.194 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:38:15.194 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 36'5 last_divergent_update: 36'5 2024-08-18T21:38:15.195 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-08-18T21:38:15.195 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.0.log:2024-08-18T21:37:44.898+0000 7f897bd1e640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-08-18T21:38:15.196 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-08-18T21:38:15.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-18T21:38:15.215 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-08-18T21:38:15.217 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-18T21:38:15.232 INFO:tasks.workunit.client.0.smithi099.stdout:success 2024-08-18T21:38:15.232 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-08-18T21:38:15.232 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.27423.out 2024-08-18T21:38:15.233 INFO:tasks.workunit.client.0.smithi099.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.27423.out': No such file or directory 2024-08-18T21:38:15.233 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-08-18T21:38:15.233 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-08-18T21:38:15.233 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-08-18T21:38:15.830 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'test' does not exist 2024-08-18T21:38:15.846 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-08-18T21:38:15.847 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:38:15.847 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:38:15.847 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:38:15.847 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:38:15.847 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:38:21.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:38:21.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-08-18T21:38:21.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-18T21:38:21.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:38:21.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-18T21:38:21.167 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:38:21.167 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:38:21.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:38:21.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:38:21.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:38:21.175 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:38:21.175 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:38:21.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:38:21.176 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:38:21.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:38:21.177 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:38:21.178 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:38:21.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:38:21.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:38:21.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:38:21.179 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:38:21.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:38:21.181 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:38:21.181 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-18T21:38:21.215 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:38:21.215 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:38:21.215 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:38:21.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27423 2024-08-18T21:38:21.217 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:38:21.217 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:38:21.217 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-08-18T21:38:21.217 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-08-18T21:38:21.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-08-18T21:38:21.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-08-18T21:38:21.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-18T21:38:21.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:38:21.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-18T21:38:21.218 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:38:21.218 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:38:21.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:38:21.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:38:21.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:38:21.219 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:38:21.219 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:38:21.220 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:38:21.221 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:38:21.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:38:21.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:38:21.222 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:38:21.222 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:38:21.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:38:21.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:38:21.223 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:38:21.224 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:38:21.225 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:38:21.225 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-18T21:38:21.226 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:38:21.226 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:38:21.226 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:38:21.227 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27423 2024-08-18T21:38:21.227 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:38:21.227 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:38:21.227 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-08-18T21:38:21.229 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-18T21:38:21.229 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:38:21.229 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:38:21.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.27423 2024-08-18T21:38:21.231 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-18T21:38:21.231 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-18T21:38:21.231 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-18T21:38:21.231 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-08-18T21:38:21.231 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-08-18T21:38:21.231 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-08-18T21:38:21.231 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-08-18T21:38:21.231 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-08-18T21:38:21.231 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-08-18T21:38:21.232 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-08-18T21:38:21.232 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-08-18T21:38:21.233 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-08-18T21:38:21.233 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:38:21.233 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:38:21.233 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-08-18T21:38:21.233 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-08-18T21:38:21.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-18T21:38:21.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-18T21:38:21.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-18T21:38:21.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-08-18T21:38:21.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-08-18T21:38:21.702 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-18T21:38:21.702 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:38:21.703 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:38:21.703 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:38:21.703 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:38:21.703 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:38:21.703 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:38:21.703 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-18T21:38:21.938 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-18T21:38:21.939 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-18T21:38:21.939 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:38:21.940 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:38:21.940 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-18T21:38:21.940 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-18T21:38:21.940 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:38:21.940 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:38:21.941 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:38:21.941 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:38:21.941 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:38:21.942 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:38:21.942 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-mon.a.asok 2024-08-18T21:38:21.942 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:38:21.942 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27423/ceph-mon.a.asok config get fsid 2024-08-18T21:38:22.058 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-18T21:38:22.058 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:38:22.058 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:38:22.058 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-18T21:38:22.059 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-18T21:38:22.059 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:38:22.059 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:38:22.059 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:38:22.060 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:38:22.060 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:38:22.060 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:38:22.060 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-mon.a.asok 2024-08-18T21:38:22.060 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:38:22.060 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.27423/ceph-mon.a.asok config get mon_host 2024-08-18T21:38:22.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-08-18T21:38:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-08-18T21:38:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-18T21:38:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-18T21:38:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-18T21:38:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-08-18T21:38:22.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-18T21:38:23.130 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-18T21:38:23.131 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:38:23.131 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:38:23.131 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:38:23.131 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:38:23.131 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:38:23.131 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:38:23.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:38:23.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:38:23.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-08-18T21:38:23.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-08-18T21:38:23.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:38:23.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:38:23.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:38:23.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:38:23.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:38:23.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:38:23.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:38:23.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:38:23.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:38:23.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-08-18T21:38:23.207 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:38:23.207 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0d22869-f4e5-4905-ba11-85fef4467cd0 2024-08-18T21:38:23.207 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b0d22869-f4e5-4905-ba11-85fef4467cd0' 2024-08-18T21:38:23.207 INFO:tasks.workunit.client.0.smithi099.stdout:add osd0 b0d22869-f4e5-4905-ba11-85fef4467cd0 2024-08-18T21:38:23.208 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:38:23.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPacJm/j71DRAAJZkFKG58IF7Z5QdvyMG9kw== 2024-08-18T21:38:23.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPacJm/j71DRAAJZkFKG58IF7Z5QdvyMG9kw=="}' 2024-08-18T21:38:23.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0d22869-f4e5-4905-ba11-85fef4467cd0 -i td/divergent-priors/0/new.json 2024-08-18T21:38:23.542 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:38:23.558 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-08-18T21:38:23.559 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPacJm/j71DRAAJZkFKG58IF7Z5QdvyMG9kw== --osd-uuid b0d22869-f4e5-4905-ba11-85fef4467cd0 2024-08-18T21:38:23.602 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:38:23.605+0000 7f9e4cfe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:38:23.609 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:38:23.613+0000 7f9e4cfe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:38:23.610 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:38:23.613+0000 7f9e4cfe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:38:23.611 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:38:23.613+0000 7f9e4cfe47c0 -1 bdev(0x560a61fd7c00 td/divergent-priors/0/block) open stat got: (1) Operation not permitted 2024-08-18T21:38:23.611 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:38:23.613+0000 7f9e4cfe47c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-08-18T21:38:30.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-08-18T21:38:30.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:38:30.059 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd0 key to auth repository 2024-08-18T21:38:30.059 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-18T21:38:30.059 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:38:30.643 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-18T21:38:30.643 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:38:30.644 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:38:30.644 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:38:30.644 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:38:30.644 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:38:30.683 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:38:30.685+0000 7f780364f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:38:30.716 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:38:30.721+0000 7f780364f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:38:30.718 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:38:30.721+0000 7f780364f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:38:31.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-18T21:38:31.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:38:31.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:38:31.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:38:31.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:38:31.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:31.175 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:38:31.175 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:38:31.175 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:31.175 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:31.705 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:31.785 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:38:31.789+0000 7f780364f7c0 -1 Falling back to public interface 2024-08-18T21:38:32.706 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:32.706 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:32.707 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:38:32.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:38:32.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:32.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:33.160 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:34.162 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:34.162 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:34.162 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:38:34.163 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:38:34.163 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:34.163 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:34.575 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:38:34.577+0000 7f780364f7c0 -1 osd.0 0 log_to_monitors true 2024-08-18T21:38:34.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:35.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:35.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:35.622 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:38:35.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:38:35.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:35.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:36.079 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:37.080 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:37.080 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:37.081 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:38:37.081 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:38:37.081 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:37.081 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:37.675 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:38.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:38.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:38.677 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:38:38.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:38:38.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:38.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:39.140 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:40.142 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:40.142 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:40.142 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:38:40.143 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:38:40.143 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:40.143 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:40.755 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:41.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:41.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:41.757 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:38:41.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:38:41.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:41.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:42.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:43.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:43.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:43.258 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:38:43.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:38:43.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:43.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:43.856 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:44.857 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:44.857 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:44.858 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:38:44.858 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:38:44.858 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:44.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:45.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:46.313 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:46.313 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:46.314 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:38:46.314 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:38:46.314 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:46.314 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:46.881 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:47.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:47.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:47.883 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:38:47.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:38:47.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:47.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:48.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:49.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:49.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:49.781 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:38:49.782 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:38:49.782 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:49.782 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:50.398 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:51.400 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:51.400 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:51.401 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:38:51.401 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:38:51.401 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:51.401 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:52.251 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:53.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:53.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:53.253 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:38:53.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:38:53.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:53.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:54.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:55.007 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:55.007 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:55.007 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:38:55.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:38:55.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:55.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:55.956 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:56.958 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:56.958 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:56.959 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:38:56.959 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:38:56.959 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:56.959 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:58.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:38:59.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:38:59.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:38:59.011 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:38:59.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:38:59.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:38:59.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:38:59.659 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:00.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:00.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:00.661 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:39:00.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:39:00.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:00.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:39:01.126 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/651436989,v1:127.0.0.1:6803/651436989] [v2:127.0.0.1:6804/651436989,v1:127.0.0.1:6805/651436989] exists,up b0d22869-f4e5-4905-ba11-85fef4467cd0 2024-08-18T21:39:01.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:39:01.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:39:01.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:39:01.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-08-18T21:39:01.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-08-18T21:39:01.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:39:01.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:39:01.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-18T21:39:01.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:39:01.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-08-18T21:39:01.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:39:01.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:39:01.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:39:01.131 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:39:01.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-08-18T21:39:01.133 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:39:01.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cf5d0225-bc36-4ab3-94ae-5d9e0c83d536 2024-08-18T21:39:01.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cf5d0225-bc36-4ab3-94ae-5d9e0c83d536' 2024-08-18T21:39:01.135 INFO:tasks.workunit.client.0.smithi099.stdout:add osd1 cf5d0225-bc36-4ab3-94ae-5d9e0c83d536 2024-08-18T21:39:01.135 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:39:01.163 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD1acJmUSnuCRAAkFyTPTJeDqouU+rw3AR15g== 2024-08-18T21:39:01.163 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD1acJmUSnuCRAAkFyTPTJeDqouU+rw3AR15g=="}' 2024-08-18T21:39:01.163 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cf5d0225-bc36-4ab3-94ae-5d9e0c83d536 -i td/divergent-priors/1/new.json 2024-08-18T21:39:01.759 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:39:01.776 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-08-18T21:39:01.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD1acJmUSnuCRAAkFyTPTJeDqouU+rw3AR15g== --osd-uuid cf5d0225-bc36-4ab3-94ae-5d9e0c83d536 2024-08-18T21:39:01.825 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:01.828+0000 7f834ba017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:01.853 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:01.856+0000 7f834ba017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:01.855 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:01.856+0000 7f834ba017c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:01.855 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:01.856+0000 7f834ba017c0 -1 bdev(0x55b9a1046800 td/divergent-priors/1/block) open stat got: (1) Operation not permitted 2024-08-18T21:39:01.856 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:01.856+0000 7f834ba017c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-08-18T21:39:08.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-08-18T21:39:08.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:39:08.047 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd1 key to auth repository 2024-08-18T21:39:08.048 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-18T21:39:08.048 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:39:08.638 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-18T21:39:08.638 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:39:08.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:39:08.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:39:08.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:39:08.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:39:08.678 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:08.680+0000 7f97141ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:08.707 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:08.708+0000 7f97141ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:08.708 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:08.712+0000 7f97141ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:09.090 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-18T21:39:09.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:39:09.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:39:09.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:39:09.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:39:09.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:09.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:39:09.091 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:39:09.092 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:09.092 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:09.532 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:09.991 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:09.992+0000 7f97141ee7c0 -1 Falling back to public interface 2024-08-18T21:39:10.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:10.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:10.535 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:39:10.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:39:10.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:10.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:11.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:12.007 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:12.007 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:12.007 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:39:12.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:39:12.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:12.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:12.487 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:12.964 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:12.968+0000 7f97141ee7c0 -1 osd.1 0 log_to_monitors true 2024-08-18T21:39:13.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:13.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:13.489 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:39:13.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:39:13.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:13.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:14.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:15.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:15.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:15.308 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:39:15.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:39:15.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:15.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:15.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:16.897 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:16.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:16.983 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:39:16.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:39:16.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:16.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:17.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:18.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:18.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:18.369 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:39:18.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:39:18.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:18.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:19.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:20.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:20.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:20.045 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:39:20.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:39:20.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:20.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:20.598 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:21.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:21.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:21.600 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:39:21.601 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:39:21.601 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:21.601 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:22.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:23.181 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:23.181 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:23.181 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:39:23.182 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:39:23.182 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:23.182 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:23.668 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:24.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:24.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:24.670 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:39:24.670 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:39:24.670 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:24.670 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:25.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:26.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:26.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:26.167 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:39:26.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:39:26.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:26.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:26.694 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:27.696 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:27.696 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:27.696 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:39:27.697 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:39:27.697 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:27.697 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:28.815 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:29.817 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:29.817 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:29.818 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:39:29.818 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:39:29.818 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:29.818 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:30.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:31.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:31.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:31.619 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:39:31.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:39:31.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:31.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:32.486 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:33.488 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:33.488 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:33.489 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:39:33.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:39:33.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:33.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:34.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:35.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:35.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:35.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:39:35.258 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:39:35.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:35.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:36.063 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:37.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:37.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:37.066 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:39:37.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:39:37.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:37.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:37.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:38.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:38.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:38.798 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:39:38.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:39:38.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:38.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:39:39.250 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2915049588,v1:127.0.0.1:6811/2915049588] [v2:127.0.0.1:6812/2915049588,v1:127.0.0.1:6813/2915049588] exists,up cf5d0225-bc36-4ab3-94ae-5d9e0c83d536 2024-08-18T21:39:39.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:39:39.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:39:39.255 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:39:39.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-08-18T21:39:39.257 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:39:39.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a517c788-2442-48b9-984a-35f904f61dbf 2024-08-18T21:39:39.257 INFO:tasks.workunit.client.0.smithi099.stdout:add osd2 a517c788-2442-48b9-984a-35f904f61dbf 2024-08-18T21:39:39.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a517c788-2442-48b9-984a-35f904f61dbf' 2024-08-18T21:39:39.258 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:39:39.287 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbasJmLl1ZERAA2AzbDzeFbfGS8h3m4VcFsg== 2024-08-18T21:39:39.287 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbasJmLl1ZERAA2AzbDzeFbfGS8h3m4VcFsg=="}' 2024-08-18T21:39:39.287 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a517c788-2442-48b9-984a-35f904f61dbf -i td/divergent-priors/2/new.json 2024-08-18T21:39:39.884 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:39:39.900 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-08-18T21:39:39.901 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbasJmLl1ZERAA2AzbDzeFbfGS8h3m4VcFsg== --osd-uuid a517c788-2442-48b9-984a-35f904f61dbf 2024-08-18T21:39:39.943 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:39.943+0000 7fa73f6417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:39.981 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:39.983+0000 7fa73f6417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:39.983 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:39.983+0000 7fa73f6417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:39.983 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:39.983+0000 7fa73f6417c0 -1 bdev(0x5618039afc00 td/divergent-priors/2/block) open stat got: (1) Operation not permitted 2024-08-18T21:39:39.984 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:39.983+0000 7fa73f6417c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-08-18T21:39:46.127 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-08-18T21:39:46.128 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:39:46.129 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd2 key to auth repository 2024-08-18T21:39:46.129 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-18T21:39:46.129 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:39:46.734 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:39:46.735 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-18T21:39:46.735 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:39:46.735 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:39:46.735 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:39:46.735 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:39:46.777 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:46.779+0000 7f381ee0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:46.780 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:46.783+0000 7f381ee0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:46.781 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:46.783+0000 7f381ee0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:39:47.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-18T21:39:47.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:39:47.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:39:47.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:39:47.159 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:39:47.160 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:39:47.160 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:47.160 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:39:47.160 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:47.160 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:39:47.604 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:48.251 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:48.251+0000 7f381ee0d7c0 -1 Falling back to public interface 2024-08-18T21:39:48.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:48.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:48.607 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:39:48.607 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:39:48.608 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:48.608 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:39:49.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:50.047 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:50.048 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:50.048 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:39:50.048 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:39:50.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:50.049 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:39:50.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:51.165 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:39:51.167+0000 7f381ee0d7c0 -1 osd.2 0 log_to_monitors true 2024-08-18T21:39:51.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:51.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:51.483 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:39:51.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:39:51.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:51.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:39:52.093 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:53.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:53.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:53.095 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:39:53.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:39:53.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:53.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:39:53.575 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:54.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:54.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:54.577 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:39:54.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:39:54.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:54.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:39:55.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:56.138 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:56.138 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:56.138 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:39:56.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:39:56.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:56.139 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:39:56.679 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:57.681 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:57.681 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:57.681 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:39:57.682 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:39:57.682 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:57.682 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:39:58.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:39:59.335 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:39:59.335 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:39:59.336 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:39:59.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:39:59.337 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:39:59.337 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:39:59.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:40:00.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:40:00.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:40:00.787 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:40:00.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:40:00.788 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:40:00.788 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:40:01.299 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:40:02.301 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:40:02.301 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:40:02.302 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:40:02.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:40:02.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:40:02.303 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:40:02.822 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:40:03.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:40:03.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:40:03.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:40:03.825 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:40:03.825 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:40:03.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:40:04.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:40:05.591 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:40:05.592 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:40:05.592 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:40:05.592 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:40:05.592 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:40:05.593 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:40:06.162 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:40:07.163 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:40:07.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:40:07.164 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:40:07.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:40:07.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:40:07.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:40:08.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:40:09.077 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:40:09.077 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:40:09.078 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:40:09.078 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:40:09.078 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:40:09.078 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:40:09.824 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:40:10.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:40:10.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:40:10.826 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:40:10.827 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:40:10.827 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:40:10.827 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:40:11.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:40:12.953 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:40:12.953 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:40:12.954 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:40:12.954 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:40:12.954 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:40:12.954 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:40:14.089 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:40:15.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:40:15.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:40:15.091 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:40:15.092 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:40:15.092 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:40:15.092 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:40:15.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:40:16.740 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:40:16.740 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:40:16.740 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:40:16.741 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:40:16.741 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:40:16.741 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:40:17.234 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3799093468,v1:127.0.0.1:6819/3799093468] [v2:127.0.0.1:6820/3799093468,v1:127.0.0.1:6821/3799093468] exists,up a517c788-2442-48b9-984a-35f904f61dbf 2024-08-18T21:40:17.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:40:17.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:40:17.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:40:17.235 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-08-18T21:40:17.961 INFO:tasks.workunit.client.0.smithi099.stderr:noout is set 2024-08-18T21:40:17.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-08-18T21:40:18.890 INFO:tasks.workunit.client.0.smithi099.stderr:noin is set 2024-08-18T21:40:18.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-08-18T21:40:19.810 INFO:tasks.workunit.client.0.smithi099.stderr:nodown is set 2024-08-18T21:40:19.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-08-18T21:40:19.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: local pool_name=test 2024-08-18T21:40:19.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: shift 2024-08-18T21:40:19.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: local allow_overwrites=true 2024-08-18T21:40:19.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: create_ec_pool: shift 2024-08-18T21:40:19.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2457: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-08-18T21:40:20.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2459: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-08-18T21:40:20.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-08-18T21:40:21.504 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'test' already exists 2024-08-18T21:40:21.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-18T21:40:22.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: create_ec_pool: '[' true = true ']' 2024-08-18T21:40:22.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2462: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-08-18T21:40:23.810 INFO:tasks.workunit.client.0.smithi099.stderr:set pool 1 allow_ec_overwrites to true 2024-08-18T21:40:23.834 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: wait_for_clean 2024-08-18T21:40:23.834 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:40:23.834 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:40:23.834 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:40:23.835 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-18T21:40:23.836 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:40:23.836 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:40:23.836 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:40:23.836 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:40:23.836 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:40:23.954 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-18T21:40:23.954 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:40:23.954 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:40:23.954 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:40:23.954 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:40:23.955 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:40:24.467 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:40:24.467 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:40:24.467 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:40:24.467 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:40:24.468 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:40:24.468 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:40:24.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836503 2024-08-18T21:40:24.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836503 2024-08-18T21:40:24.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836503' 2024-08-18T21:40:24.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:40:24.652 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:40:24.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672980 2024-08-18T21:40:24.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672980 2024-08-18T21:40:24.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836503 1-42949672980' 2024-08-18T21:40:24.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:40:24.802 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:40:24.963 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509450 2024-08-18T21:40:24.963 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509450 2024-08-18T21:40:24.963 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509450' 2024-08-18T21:40:24.963 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:40:24.964 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836503 2024-08-18T21:40:24.964 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:40:24.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:40:24.965 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836503 2024-08-18T21:40:24.965 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:40:24.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836503 2024-08-18T21:40:24.967 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836503 2024-08-18T21:40:24.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-08-18T21:40:24.967 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:40:25.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836501 -lt 21474836503 2024-08-18T21:40:25.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:40:26.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:40:26.411 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:40:26.864 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836504 -lt 21474836503 2024-08-18T21:40:26.864 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:40:26.865 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672980 2024-08-18T21:40:26.865 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:40:26.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:40:26.868 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672980 2024-08-18T21:40:26.868 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:40:26.869 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672980 2024-08-18T21:40:26.869 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-08-18T21:40:26.870 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672980 2024-08-18T21:40:26.870 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:40:27.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672980 -lt 42949672980 2024-08-18T21:40:27.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:40:27.307 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-08-18T21:40:27.307 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:40:27.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:40:27.309 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509450 2024-08-18T21:40:27.309 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:40:27.310 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509450 2024-08-18T21:40:27.310 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509450 2024-08-18T21:40:27.311 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-08-18T21:40:27.311 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:40:27.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509450 -lt 64424509450 2024-08-18T21:40:27.773 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:40:27.774 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:40:27.774 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:40:28.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:40:28.345 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:40:28.345 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:40:28.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:40:28.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:40:28.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:40:28.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:40:28.346 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:40:28.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-18T21:40:28.753 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:40:28.754 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:40:28.754 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:40:29.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-18T21:40:29.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:40:29.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:40:29.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2466: create_ec_pool: return 0 2024-08-18T21:40:29.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-08-18T21:40:29.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:40:29.307 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:40:29.737 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:40:29.737 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:40:29.737 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:40:29.737 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:40:29.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:40:29.738 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:40:29.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836506 2024-08-18T21:40:29.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836506 2024-08-18T21:40:29.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836506' 2024-08-18T21:40:29.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:40:29.898 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:40:30.056 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672983 2024-08-18T21:40:30.056 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672983 2024-08-18T21:40:30.056 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836506 1-42949672983' 2024-08-18T21:40:30.056 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:40:30.056 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:40:30.207 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509453 2024-08-18T21:40:30.207 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509453 2024-08-18T21:40:30.207 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836506 1-42949672983 2-64424509453' 2024-08-18T21:40:30.207 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:40:30.208 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-08-18T21:40:30.208 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:40:30.208 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:40:30.210 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:40:30.210 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836506 2024-08-18T21:40:30.210 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836506 2024-08-18T21:40:30.211 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836506 2024-08-18T21:40:30.211 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-08-18T21:40:30.211 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:40:30.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836505 -lt 21474836506 2024-08-18T21:40:30.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:40:31.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:40:31.619 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:40:32.098 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836507 -lt 21474836506 2024-08-18T21:40:32.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:40:32.100 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-08-18T21:40:32.100 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:40:32.101 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:40:32.102 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672983 2024-08-18T21:40:32.102 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:40:32.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672983 2024-08-18T21:40:32.103 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672983 2024-08-18T21:40:32.104 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-08-18T21:40:32.104 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:40:32.537 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672983 -lt 42949672983 2024-08-18T21:40:32.538 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:40:32.538 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509453 2024-08-18T21:40:32.539 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:40:32.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:40:32.541 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509453 2024-08-18T21:40:32.541 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:40:32.542 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509453 2024-08-18T21:40:32.542 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-08-18T21:40:32.543 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509453 2024-08-18T21:40:32.543 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:40:32.991 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509454 -lt 64424509453 2024-08-18T21:40:32.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-08-18T21:40:32.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:40:32.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:40:32.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:40:32.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-18T21:40:32.992 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:40:32.993 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:40:32.993 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:40:32.993 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:40:32.993 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:40:33.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-18T21:40:33.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:40:33.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:40:33.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:40:33.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:40:33.096 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:40:33.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:40:33.521 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:40:33.521 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:40:33.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:40:33.522 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:40:33.522 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:40:33.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836509 2024-08-18T21:40:33.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836509 2024-08-18T21:40:33.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836509' 2024-08-18T21:40:33.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:40:33.689 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:40:33.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672986 2024-08-18T21:40:33.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672986 2024-08-18T21:40:33.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836509 1-42949672986' 2024-08-18T21:40:33.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:40:33.835 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:40:34.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509456 2024-08-18T21:40:34.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509456 2024-08-18T21:40:34.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836509 1-42949672986 2-64424509456' 2024-08-18T21:40:34.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:40:34.009 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836509 2024-08-18T21:40:34.009 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:40:34.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:40:34.010 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836509 2024-08-18T21:40:34.010 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:40:34.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836509 2024-08-18T21:40:34.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-08-18T21:40:34.011 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836509 2024-08-18T21:40:34.011 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:40:34.426 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836509 -lt 21474836509 2024-08-18T21:40:34.426 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:40:34.427 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672986 2024-08-18T21:40:34.427 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:40:34.428 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:40:34.429 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672986 2024-08-18T21:40:34.429 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:40:34.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672986 2024-08-18T21:40:34.431 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672986 2024-08-18T21:40:34.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-08-18T21:40:34.431 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:40:34.863 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672986 -lt 42949672986 2024-08-18T21:40:34.863 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:40:34.865 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509456 2024-08-18T21:40:34.865 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:40:34.866 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:40:34.867 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509456 2024-08-18T21:40:34.867 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:40:34.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509456 2024-08-18T21:40:34.869 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509456 2024-08-18T21:40:34.869 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-08-18T21:40:34.869 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:40:35.317 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509454 -lt 64424509456 2024-08-18T21:40:35.317 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:40:36.318 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:40:36.319 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:40:36.747 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509456 -lt 64424509456 2024-08-18T21:40:36.747 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:40:36.748 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:40:36.748 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:40:37.332 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:40:37.332 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:40:37.333 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:40:37.333 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:40:37.333 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:40:37.333 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:40:37.333 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:40:37.334 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:40:37.740 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-18T21:40:37.740 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:40:37.741 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:40:37.741 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:40:38.292 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-18T21:40:38.292 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:40:38.292 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:40:38.293 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-08-18T21:40:38.293 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-08-18T21:40:38.752 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:40:38.769 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-08-18T21:40:38.770 INFO:tasks.workunit.client.0.smithi099.stdout:primary and soon to be divergent is 1 2024-08-18T21:40:38.770 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-08-18T21:40:38.770 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:40:39.184 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:40:39.184 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-08-18T21:40:23.147249+0000 0'0 29:18 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:32:22.488254+0000 0 0 2024-08-18T21:40:39.184 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:40:39.184 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:40:39.185 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:40:39.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-08-18T21:40:39.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-08-18T21:40:39.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-08-18T21:40:39.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-08-18T21:40:39.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-08-18T21:40:39.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-08-18T21:40:39.201 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-08-18T21:40:39.201 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-08-18T21:40:39.201 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-08-18T21:40:39.201 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-08-18T21:40:39.201 INFO:tasks.workunit.client.0.smithi099.stdout:writing initial objects 2024-08-18T21:40:39.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-08-18T21:40:39.202 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-08-18T21:40:39.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:39.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-08-18T21:40:39.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:39.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-08-18T21:40:39.790 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:39.790 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-08-18T21:40:40.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:40.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-08-18T21:40:40.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:40.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-08-18T21:40:40.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:40.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-08-18T21:40:40.727 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:40.728 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-08-18T21:40:40.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:40.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-08-18T21:40:41.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:41.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-08-18T21:40:41.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:41.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-08-18T21:40:41.708 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:41.708 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-08-18T21:40:41.919 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:41.919 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-08-18T21:40:42.178 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:42.178 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-08-18T21:40:42.404 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:42.405 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-08-18T21:40:42.657 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:42.657 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-08-18T21:40:42.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:42.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-08-18T21:40:43.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:43.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-08-18T21:40:43.427 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:43.427 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-08-18T21:40:43.636 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:43.636 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-08-18T21:40:43.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:43.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-08-18T21:40:44.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:44.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-08-18T21:40:44.392 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:44.392 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-08-18T21:40:44.735 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:44.735 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-08-18T21:40:44.993 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:44.994 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-08-18T21:40:45.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:45.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-08-18T21:40:45.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:45.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-08-18T21:40:45.740 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:45.740 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-08-18T21:40:45.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:45.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-08-18T21:40:46.243 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:46.243 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-08-18T21:40:46.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:46.480 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-08-18T21:40:46.671 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:46.671 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-08-18T21:40:46.931 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:46.931 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-08-18T21:40:47.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:47.200 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-08-18T21:40:47.442 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:47.443 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-08-18T21:40:47.702 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:47.702 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-08-18T21:40:47.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:47.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-08-18T21:40:48.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:48.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-08-18T21:40:48.428 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:48.428 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-08-18T21:40:48.667 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:48.667 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-08-18T21:40:48.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:48.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-08-18T21:40:49.128 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:49.128 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-08-18T21:40:49.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:49.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-08-18T21:40:49.574 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:49.594 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-08-18T21:40:49.815 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:49.815 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-08-18T21:40:50.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:50.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-08-18T21:40:50.318 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:50.318 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-08-18T21:40:50.587 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:50.587 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-08-18T21:40:50.788 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:50.789 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-08-18T21:40:51.072 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:51.073 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-08-18T21:40:51.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:51.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-08-18T21:40:51.575 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:51.575 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-08-18T21:40:51.793 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:51.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-08-18T21:40:52.030 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:52.030 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-08-18T21:40:52.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:52.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-08-18T21:40:52.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:52.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-08-18T21:40:52.791 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:52.791 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-08-18T21:40:53.025 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:53.025 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-08-18T21:40:53.277 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:53.277 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-08-18T21:40:53.487 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:53.487 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-08-18T21:40:53.722 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:53.722 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-08-18T21:40:53.990 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:53.990 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-08-18T21:40:54.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:54.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-08-18T21:40:54.476 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:54.476 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-08-18T21:40:54.694 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:54.695 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-08-18T21:40:54.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:54.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-08-18T21:40:55.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:55.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-08-18T21:40:55.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:55.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-08-18T21:40:55.633 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:55.634 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-08-18T21:40:55.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:55.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-08-18T21:40:56.128 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:56.128 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-08-18T21:40:56.354 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:56.355 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-08-18T21:40:56.634 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:56.634 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-08-18T21:40:56.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:56.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-08-18T21:40:57.077 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:57.077 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-08-18T21:40:57.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:57.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-08-18T21:40:57.530 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:57.530 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-08-18T21:40:57.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:57.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-08-18T21:40:58.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:58.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-08-18T21:40:58.325 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:58.326 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-08-18T21:40:58.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:58.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-08-18T21:40:58.837 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:58.837 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-08-18T21:40:59.054 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:59.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-08-18T21:40:59.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:59.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-08-18T21:40:59.548 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:59.548 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-08-18T21:40:59.759 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:40:59.759 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-08-18T21:41:00.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:00.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-08-18T21:41:00.244 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:00.244 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-08-18T21:41:00.455 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:00.455 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-08-18T21:41:00.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:00.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-08-18T21:41:00.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:00.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-08-18T21:41:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:01.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-08-18T21:41:01.401 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:01.401 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-08-18T21:41:01.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:01.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-08-18T21:41:01.896 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:01.896 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-08-18T21:41:02.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:02.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-08-18T21:41:02.398 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:02.398 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-08-18T21:41:02.634 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:02.634 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-08-18T21:41:02.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:02.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-08-18T21:41:03.161 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:03.161 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-08-18T21:41:03.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-08-18T21:41:03.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-08-18T21:41:03.605 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-18T21:41:03.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-08-18T21:41:03.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:41:03.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:41:03.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:41:03.606 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-18T21:41:03.606 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:41:03.607 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:41:03.607 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:41:03.607 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:41:03.607 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:41:03.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-18T21:41:03.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:41:03.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:41:03.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:41:03.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:41:03.685 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:41:04.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:41:04.151 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:41:04.151 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:41:04.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:41:04.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:41:04.152 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:41:04.338 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=21474836517 2024-08-18T21:41:04.339 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 21474836517 2024-08-18T21:41:04.339 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836517' 2024-08-18T21:41:04.339 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:41:04.339 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:41:04.501 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=42949672994 2024-08-18T21:41:04.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 42949672994 2024-08-18T21:41:04.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836517 1-42949672994' 2024-08-18T21:41:04.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:41:04.502 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:41:04.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=64424509464 2024-08-18T21:41:04.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 64424509464 2024-08-18T21:41:04.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-21474836517 1-42949672994 2-64424509464' 2024-08-18T21:41:04.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:41:04.664 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836517 2024-08-18T21:41:04.664 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:41:04.664 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:41:04.665 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-21474836517 2024-08-18T21:41:04.665 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:41:04.666 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 21474836517 2024-08-18T21:41:04.667 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=21474836517 2024-08-18T21:41:04.667 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 21474836517' 2024-08-18T21:41:04.667 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:41:05.081 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836515 -lt 21474836517 2024-08-18T21:41:05.081 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:41:06.083 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:41:06.083 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:41:06.542 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 21474836518 -lt 21474836517 2024-08-18T21:41:06.543 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:41:06.543 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672994 2024-08-18T21:41:06.543 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:41:06.545 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:41:06.545 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-42949672994 2024-08-18T21:41:06.546 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:41:06.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=42949672994 2024-08-18T21:41:06.547 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 42949672994 2024-08-18T21:41:06.548 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-08-18T21:41:06.548 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:41:07.019 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 42949672994 -lt 42949672994 2024-08-18T21:41:07.019 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:41:07.020 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:41:07.020 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509464 2024-08-18T21:41:07.021 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:41:07.022 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-64424509464 2024-08-18T21:41:07.022 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:41:07.023 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=64424509464 2024-08-18T21:41:07.023 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 64424509464 2024-08-18T21:41:07.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-08-18T21:41:07.024 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:41:07.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 64424509464 -lt 64424509464 2024-08-18T21:41:07.530 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:41:07.531 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:41:07.531 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:41:08.170 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:41:08.171 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:41:08.171 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:41:08.172 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:41:08.172 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:41:08.172 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:41:08.172 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:41:08.172 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:41:08.586 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-18T21:41:08.587 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:41:08.588 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:41:08.588 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:41:09.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-18T21:41:09.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:41:09.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:41:09.154 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-08-18T21:41:09.154 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-08-18T21:41:09.154 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-08-18T21:41:09.154 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-08-18T21:41:09.155 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-08-18T21:41:09.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-08-18T21:41:09.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-08-18T21:41:09.624 INFO:tasks.workunit.client.0.smithi099.stdout:blackholing osds 0 2 2024-08-18T21:41:09.625 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:41:10.042 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:41:10.042 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-08-18T21:40:23.147249+0000 29'100 29:118 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-20T04:32:22.488254+0000 0 0 2024-08-18T21:41:10.042 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:41:10.042 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:41:10.043 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:41:10.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:41:10.058 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-08-18T21:41:10.058 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-18T21:41:10.058 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-18T21:41:10.059 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:41:10.059 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:41:10.059 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:41:10.059 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.0.asok 2024-08-18T21:41:10.059 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-08-18T21:41:10.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.0.asok config set objectstore_blackhole 1 2024-08-18T21:41:10.154 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:41:10.155 INFO:tasks.workunit.client.0.smithi099.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-18T21:41:10.155 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:41:10.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:41:10.167 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-08-18T21:41:10.167 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-18T21:41:10.167 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-18T21:41:10.168 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:41:10.168 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:41:10.168 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:41:10.168 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.2.asok 2024-08-18T21:41:10.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-08-18T21:41:10.168 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.2.asok config set objectstore_blackhole 1 2024-08-18T21:41:10.265 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:41:10.265 INFO:tasks.workunit.client.0.smithi099.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-08-18T21:41:10.265 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:41:10.277 INFO:tasks.workunit.client.0.smithi099.stdout:writing divergent object 2024-08-18T21:41:10.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-08-18T21:41:10.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-08-18T21:41:10.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-08-18T21:41:11.280 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-08-18T21:41:11.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-08-18T21:41:11.528 INFO:tasks.workunit.client.0.smithi099.stdout:created pool test snap snap1 2024-08-18T21:41:11.532 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-08-18T21:41:11.533 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-08-18T21:41:11.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-08-18T21:41:21.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-08-18T21:41:21.538 INFO:tasks.workunit.client.0.smithi099.stderr:rados: no process found 2024-08-18T21:41:21.539 INFO:tasks.workunit.client.0.smithi099.stdout:killing all the osds 2024-08-18T21:41:21.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-08-18T21:41:21.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:41:21.950 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:41:21.951 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 active+clean 2024-08-18T21:40:23.147249+0000 30'104 30:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-19T23:26:56.850851+0000 0 0 2024-08-18T21:41:21.951 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:41:21.951 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:41:21.951 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:41:21.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-08-18T21:41:21.967 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:41:21.967 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:41:21.968 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:41:21.968 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:41:21.968 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:41:22.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:41:22.077 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-08-18T21:41:22.077 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-08-18T21:41:22.716 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already down. 2024-08-18T21:41:22.733 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-08-18T21:41:22.733 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-08-18T21:41:23.350 INFO:tasks.workunit.client.0.smithi099.stderr:osd.1 is already down. 2024-08-18T21:41:23.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-08-18T21:41:23.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-08-18T21:41:24.016 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already down. 2024-08-18T21:41:24.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:41:24.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-08-18T21:41:24.604 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already out. 2024-08-18T21:41:24.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:41:24.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-08-18T21:41:25.245 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already out. 2024-08-18T21:41:25.262 INFO:tasks.workunit.client.0.smithi099.stdout:bringing up non_divergent 0 2 2024-08-18T21:41:25.262 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-08-18T21:41:25.262 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:41:25.691 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:41:25.691 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 stale+active+clean 2024-08-18T21:40:23.147249+0000 30'104 30:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-19T23:26:56.850851+0000 0 0 2024-08-18T21:41:25.691 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:41:25.692 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:41:25.692 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:41:25.710 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:41:25.710 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:41:25.711 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:41:25.712 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:41:25.712 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:41:25.712 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:41:25.712 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:41:25.712 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:41:25.712 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:41:25.712 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:41:25.712 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:41:25.713 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:41:25.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:41:25.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:41:25.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:41:25.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:41:25.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:41:25.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:41:25.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:41:25.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:41:25.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:41:25.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-18T21:41:25.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-18T21:41:25.715 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:41:25.715 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:41:25.716 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-18T21:41:25.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-18T21:41:25.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:41:25.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:41:25.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:41:25.748 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:41:25.749+0000 7fb4bea3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:41:25.750 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:41:25.749+0000 7fb4bea3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:41:25.751 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:41:25.753+0000 7fb4bea3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:41:26.144 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-18T21:41:26.144 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:41:26.144 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:41:26.145 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:41:26.145 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:41:26.145 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:41:26.145 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:41:26.146 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:41:26.146 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:41:26.146 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:41:26.594 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:41:26.738 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:41:26.737+0000 7fb4bea3b7c0 -1 Falling back to public interface 2024-08-18T21:41:27.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:41:27.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:41:27.596 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:41:27.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:41:27.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:41:27.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:41:28.063 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:41:29.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:41:29.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:41:29.065 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:41:29.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:41:29.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:41:29.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:41:29.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:41:30.164 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:41:30.165+0000 7fb4bea3b7c0 -1 osd.0 29 log_to_monitors true 2024-08-18T21:41:30.512 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:41:30.512 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:41:30.512 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:41:30.513 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:41:30.513 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:41:30.513 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:41:31.010 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/420852772,v1:127.0.0.1:6803/420852772] [v2:127.0.0.1:6804/420852772,v1:127.0.0.1:6805/420852772] exists,up b0d22869-f4e5-4905-ba11-85fef4467cd0 2024-08-18T21:41:31.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:41:31.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:41:31.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:41:31.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:41:31.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-08-18T21:41:31.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:41:31.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:41:31.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-18T21:41:31.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:41:31.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-08-18T21:41:31.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:41:31.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:41:31.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:41:31.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:41:31.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-18T21:41:31.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-18T21:41:31.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:41:31.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:41:31.015 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:41:31.016 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:41:31.016 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:41:31.016 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:41:31.016 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:41:31.016 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:41:31.016 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:41:31.016 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:41:31.016 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:41:31.016 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:41:31.016 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:41:31.016 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:41:31.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:41:31.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:41:31.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:41:31.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:41:31.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:41:31.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:41:31.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-08-18T21:41:31.018 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-18T21:41:31.018 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:41:31.019 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:41:31.019 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-08-18T21:41:31.019 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-18T21:41:31.019 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:41:31.019 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:41:31.019 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:41:31.050 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:41:31.049+0000 7fe30f1d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:41:31.053 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:41:31.053+0000 7fe30f1d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:41:31.054 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:41:31.053+0000 7fe30f1d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:41:31.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-18T21:41:31.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:41:31.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:41:31.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:41:31.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:41:31.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:41:31.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:41:31.432 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:41:31.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:41:31.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:41:31.947 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:41:32.109 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:41:32.113+0000 7fe30f1d67c0 -1 Falling back to public interface 2024-08-18T21:41:32.948 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:41:32.949 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:41:32.949 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:41:32.949 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:41:32.950 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:41:32.950 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:41:33.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:41:34.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:41:34.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:41:34.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:41:34.432 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:41:34.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:41:34.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:41:34.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:41:35.561 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:41:35.565+0000 7fe30f1d67c0 -1 osd.2 29 log_to_monitors true 2024-08-18T21:41:35.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:41:35.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:41:35.884 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:41:35.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:41:35.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:41:35.885 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:41:36.447 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/2898902377,v1:127.0.0.1:6811/2898902377] [v2:127.0.0.1:6812/2898902377,v1:127.0.0.1:6813/2898902377] exists,up a517c788-2442-48b9-984a-35f904f61dbf 2024-08-18T21:41:36.447 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:41:36.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:41:36.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:41:36.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:41:36.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-08-18T21:41:37.029 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already in. 2024-08-18T21:41:37.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:41:37.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-08-18T21:41:38.195 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already in. 2024-08-18T21:41:38.216 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-08-18T21:41:43.219 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-08-18T21:41:43.220 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-08-18T21:41:43.220 INFO:tasks.workunit.client.0.smithi099.stdout:writing non-divergent object existing_4 2024-08-18T21:41:43.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-08-18T21:41:43.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:41:43.646 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:41:43.647 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-08-18T21:41:38.658596+0000 29'100 44:130 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:03:08.844774+0000 0 0 2024-08-18T21:41:43.647 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:41:43.647 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:41:43.647 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:41:43.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-08-18T21:41:43.791 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-18T21:41:43.791 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-08-18T21:41:43.791 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:41:43.791 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:41:43.791 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:41:43.792 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-18T21:41:43.792 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:41:43.792 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:41:43.793 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:41:43.793 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:41:43.793 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:41:43.855 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-18T21:41:43.855 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:41:43.855 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:41:43.855 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:41:43.855 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:41:43.856 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:41:44.326 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:41:44.326 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:41:44.326 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:41:44.326 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:41:44.326 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:41:44.327 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:41:44.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=158913789957 2024-08-18T21:41:44.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 158913789957 2024-08-18T21:41:44.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-158913789957' 2024-08-18T21:41:44.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:41:44.519 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:41:44.637 INFO:tasks.workunit.client.0.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-18T21:41:44.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq= 2024-08-18T21:41:44.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z '' 2024-08-18T21:41:44.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: continue 2024-08-18T21:41:44.639 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:41:44.639 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:41:44.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=167503724548 2024-08-18T21:41:44.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 167503724548 2024-08-18T21:41:44.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-158913789957 2-167503724548' 2024-08-18T21:41:44.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:41:44.798 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-158913789957 2024-08-18T21:41:44.798 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:41:44.799 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:41:44.800 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-158913789957 2024-08-18T21:41:44.800 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:41:44.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=158913789957 2024-08-18T21:41:44.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 158913789957' 2024-08-18T21:41:44.802 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 158913789957 2024-08-18T21:41:44.802 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:41:45.222 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 158913789955 -lt 158913789957 2024-08-18T21:41:45.222 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:41:46.224 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:41:46.225 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:41:46.764 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 158913789958 -lt 158913789957 2024-08-18T21:41:46.764 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:41:46.765 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-167503724548 2024-08-18T21:41:46.765 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:41:46.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:41:46.768 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-167503724548 2024-08-18T21:41:46.768 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:41:46.769 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=167503724548 2024-08-18T21:41:46.770 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2024-08-18T21:41:46.770 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 167503724548 2024-08-18T21:41:46.770 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:41:47.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 167503724549 -lt 167503724548 2024-08-18T21:41:47.213 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:41:47.214 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:41:47.214 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:41:47.793 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:41:47.793 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:41:47.794 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:41:47.794 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:41:47.794 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:41:47.794 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:41:47.794 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:41:47.794 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:41:48.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:41:48.222 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:41:48.222 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:41:48.222 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:41:48.766 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:41:48.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-18T21:41:48.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-18T21:41:48.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-18T21:41:48.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:41:48.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-18T21:41:48.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:41:48.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:41:48.868 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:41:48.868 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:41:48.868 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:41:48.868 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:41:48.869 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:41:48.869 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:41:49.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:41:49.305 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:41:49.305 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:41:49.305 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:41:49.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:41:49.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:41:49.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:41:49.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:41:49.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:41:49.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:41:49.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:41:49.849 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:41:49.849 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:41:50.426 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:41:50.426 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:41:50.426 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-18T21:41:50.426 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:41:50.426 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-18T21:41:50.627 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:41:50.627 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:41:50.628 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:41:50.628 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:41:50.628 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:41:50.628 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:41:50.629 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:41:50.629 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:41:51.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:41:51.040 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:41:51.041 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:41:51.041 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:41:51.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:41:51.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:41:51.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:41:51.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:41:51.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:41:51.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:41:51.614 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:41:51.615 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:41:51.615 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:41:52.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:41:52.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:41:52.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-18T21:41:52.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:41:52.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-18T21:41:52.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:41:52.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:41:52.571 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:41:52.571 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:41:52.571 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:41:52.571 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:41:52.572 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:41:52.572 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:41:53.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:41:53.017 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:41:53.018 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:41:53.018 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:41:53.593 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:41:53.593 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:41:53.593 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:41:53.593 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:41:53.593 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:41:53.593 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:41:53.593 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:41:53.594 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:41:53.594 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:41:54.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:41:54.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:41:54.189 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-18T21:41:54.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:41:54.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-18T21:41:54.991 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:41:54.991 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:41:54.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:41:54.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:41:54.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:41:54.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:41:54.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:41:54.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:41:55.440 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:41:55.440 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:41:55.441 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:41:55.441 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:41:55.991 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:41:55.991 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:41:55.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:41:55.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:41:55.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:41:55.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:41:55.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:41:55.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:41:55.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:41:56.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:41:56.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:41:56.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-18T21:41:56.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:41:56.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-18T21:41:58.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:41:58.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:41:58.181 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:41:58.181 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:41:58.181 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:41:58.181 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:41:58.182 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:41:58.182 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:41:58.613 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:41:58.613 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:41:58.614 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:41:58.614 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:41:59.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:41:59.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:41:59.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:41:59.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:41:59.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:41:59.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:41:59.176 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:41:59.177 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:41:59.177 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:41:59.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:41:59.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:41:59.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-18T21:41:59.739 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:41:59.739 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-18T21:42:02.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:42:02.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:42:02.941 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:42:02.941 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:42:02.941 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:42:02.941 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:42:02.942 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:42:02.942 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:42:03.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:42:03.385 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:42:03.385 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:03.386 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:03.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:42:03.961 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:42:03.961 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:42:03.961 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:42:03.961 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:42:03.961 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:42:03.961 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:42:03.961 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:42:03.962 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:42:04.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:42:04.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:42:04.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-18T21:42:04.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:42:04.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-18T21:42:10.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:42:10.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:42:10.980 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:42:10.980 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:42:10.980 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:42:10.980 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:42:10.981 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:42:10.981 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:42:11.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:42:11.419 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:42:11.420 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:11.420 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:12.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:42:12.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:42:12.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:42:12.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:42:12.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:42:12.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:42:12.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:42:12.011 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:42:12.011 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:42:12.563 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:42:12.564 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:42:12.564 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-18T21:42:12.564 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:42:12.564 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-18T21:42:19.865 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:42:19.865 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:42:19.866 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:42:19.866 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:42:19.866 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:42:19.866 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:42:19.867 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:42:19.867 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:42:20.304 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:42:20.304 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:42:20.304 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:20.304 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:20.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:42:20.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:42:20.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:42:20.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:42:20.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:42:20.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:42:20.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:42:20.868 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:42:20.868 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:42:21.455 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:42:21.456 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:42:21.456 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-18T21:42:21.456 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-18T21:42:22.029 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:42:22.029 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_fingerprint": "1749aa68-d2b4-498a-a3c5-2ae6c810744e", 2024-08-18T21:42:22.029 INFO:tasks.workunit.client.0.smithi099.stdout: "version": "19.0.0-5847-g79766939", 2024-08-18T21:42:22.029 INFO:tasks.workunit.client.0.smithi099.stdout: "commit": "797669399b32d2915e64eb6d4b1257ae3c50f80b", 2024-08-18T21:42:22.029 INFO:tasks.workunit.client.0.smithi099.stdout: "timestamp": "2024-08-18T21:42:22.032832+0000", 2024-08-18T21:42:22.029 INFO:tasks.workunit.client.0.smithi099.stdout: "tag": "", 2024-08-18T21:42:22.029 INFO:tasks.workunit.client.0.smithi099.stdout: "health": { 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: "status": "HEALTH_WARN", 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: "checks": { 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: "OSDMAP_FLAGS": { 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 17 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [], 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.030 INFO:tasks.workunit.client.0.smithi099.stdout: "OSD_DOWN": { 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 osds down", 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "osd.1 (root=default,host=smithi099) is down" 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.031 INFO:tasks.workunit.client.0.smithi099.stdout: "PG_DEGRADED": { 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:42:22.032 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "application not enabled on pool 'test'" 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.033 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-18T21:42:22.034 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.034 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.034 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:42:22.034 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.034 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.034 INFO:tasks.workunit.client.0.smithi099.stdout: "mutes": [] 2024-08-18T21:42:22.034 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.035 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_first_committed": 1, 2024-08-18T21:42:22.035 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_last_committed": 1, 2024-08-18T21:42:22.035 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap": { 2024-08-18T21:42:22.035 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:42:22.035 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:42:22.035 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:38:21.266415Z", 2024-08-18T21:42:22.036 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:38:21.266415Z", 2024-08-18T21:42:22.036 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release": 19, 2024-08-18T21:42:22.036 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release_name": "squid", 2024-08-18T21:42:22.036 INFO:tasks.workunit.client.0.smithi099.stdout: "election_strategy": 1, 2024-08-18T21:42:22.036 INFO:tasks.workunit.client.0.smithi099.stdout: "disallowed_leaders: ": "", 2024-08-18T21:42:22.036 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": false, 2024-08-18T21:42:22.037 INFO:tasks.workunit.client.0.smithi099.stdout: "tiebreaker_mon": "", 2024-08-18T21:42:22.037 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_ranks: ": "", 2024-08-18T21:42:22.037 INFO:tasks.workunit.client.0.smithi099.stdout: "features": { 2024-08-18T21:42:22.037 INFO:tasks.workunit.client.0.smithi099.stdout: "persistent": [ 2024-08-18T21:42:22.037 INFO:tasks.workunit.client.0.smithi099.stdout: "kraken", 2024-08-18T21:42:22.037 INFO:tasks.workunit.client.0.smithi099.stdout: "luminous", 2024-08-18T21:42:22.037 INFO:tasks.workunit.client.0.smithi099.stdout: "mimic", 2024-08-18T21:42:22.038 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap-prune", 2024-08-18T21:42:22.038 INFO:tasks.workunit.client.0.smithi099.stdout: "nautilus", 2024-08-18T21:42:22.038 INFO:tasks.workunit.client.0.smithi099.stdout: "octopus", 2024-08-18T21:42:22.038 INFO:tasks.workunit.client.0.smithi099.stdout: "pacific", 2024-08-18T21:42:22.038 INFO:tasks.workunit.client.0.smithi099.stdout: "elector-pinging", 2024-08-18T21:42:22.039 INFO:tasks.workunit.client.0.smithi099.stdout: "quincy", 2024-08-18T21:42:22.039 INFO:tasks.workunit.client.0.smithi099.stdout: "reef", 2024-08-18T21:42:22.039 INFO:tasks.workunit.client.0.smithi099.stdout: "squid" 2024-08-18T21:42:22.039 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.039 INFO:tasks.workunit.client.0.smithi099.stdout: "optional": [] 2024-08-18T21:42:22.039 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.040 INFO:tasks.workunit.client.0.smithi099.stdout: "mons": [ 2024-08-18T21:42:22.040 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.040 INFO:tasks.workunit.client.0.smithi099.stdout: "rank": 0, 2024-08-18T21:42:22.040 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "a", 2024-08-18T21:42:22.040 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:42:22.040 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.041 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.041 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.041 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115", 2024-08-18T21:42:22.041 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 0 2024-08-18T21:42:22.041 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.041 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.042 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.042 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115/0", 2024-08-18T21:42:22.042 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-18T21:42:22.042 INFO:tasks.workunit.client.0.smithi099.stdout: "priority": 0, 2024-08-18T21:42:22.042 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 0, 2024-08-18T21:42:22.043 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_location": "{}" 2024-08-18T21:42:22.043 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.043 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.043 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.043 INFO:tasks.workunit.client.0.smithi099.stdout: "quorum": [ 2024-08-18T21:42:22.044 INFO:tasks.workunit.client.0.smithi099.stdout: 0 2024-08-18T21:42:22.044 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.044 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap": { 2024-08-18T21:42:22.044 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 44, 2024-08-18T21:42:22.044 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:42:22.044 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:38:22.293071+0000", 2024-08-18T21:42:22.045 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:41:38.272485+0000", 2024-08-18T21:42:22.045 INFO:tasks.workunit.client.0.smithi099.stdout: "last_up_change": "2024-08-18T21:41:35.834167+0000", 2024-08-18T21:42:22.045 INFO:tasks.workunit.client.0.smithi099.stdout: "last_in_change": "2024-08-18T21:41:37.569144+0000", 2024-08-18T21:42:22.045 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-18T21:42:22.045 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_num": 5800384, 2024-08-18T21:42:22.046 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_set": [ 2024-08-18T21:42:22.046 INFO:tasks.workunit.client.0.smithi099.stdout: "nodown", 2024-08-18T21:42:22.046 INFO:tasks.workunit.client.0.smithi099.stdout: "noin", 2024-08-18T21:42:22.046 INFO:tasks.workunit.client.0.smithi099.stdout: "noout", 2024-08-18T21:42:22.046 INFO:tasks.workunit.client.0.smithi099.stdout: "pglog_hardlimit", 2024-08-18T21:42:22.046 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snapdirs", 2024-08-18T21:42:22.047 INFO:tasks.workunit.client.0.smithi099.stdout: "recovery_deletes", 2024-08-18T21:42:22.047 INFO:tasks.workunit.client.0.smithi099.stdout: "sortbitwise" 2024-08-18T21:42:22.047 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.047 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_version": 10, 2024-08-18T21:42:22.047 INFO:tasks.workunit.client.0.smithi099.stdout: "full_ratio": 0.99000000953674316, 2024-08-18T21:42:22.048 INFO:tasks.workunit.client.0.smithi099.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-18T21:42:22.048 INFO:tasks.workunit.client.0.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-18T21:42:22.048 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_snapshot": "", 2024-08-18T21:42:22.048 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_max": 1, 2024-08-18T21:42:22.048 INFO:tasks.workunit.client.0.smithi099.stdout: "max_osd": 3, 2024-08-18T21:42:22.048 INFO:tasks.workunit.client.0.smithi099.stdout: "require_min_compat_client": "luminous", 2024-08-18T21:42:22.049 INFO:tasks.workunit.client.0.smithi099.stdout: "min_compat_client": "jewel", 2024-08-18T21:42:22.049 INFO:tasks.workunit.client.0.smithi099.stdout: "require_osd_release": "squid", 2024-08-18T21:42:22.049 INFO:tasks.workunit.client.0.smithi099.stdout: "allow_crimson": false, 2024-08-18T21:42:22.049 INFO:tasks.workunit.client.0.smithi099.stdout: "pools": [ 2024-08-18T21:42:22.049 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.049 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:42:22.050 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_name": "test", 2024-08-18T21:42:22.050 INFO:tasks.workunit.client.0.smithi099.stdout: "create_time": "2024-08-18T21:40:21.104981+0000", 2024-08-18T21:42:22.050 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": 16389, 2024-08-18T21:42:22.050 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-08-18T21:42:22.050 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 3, 2024-08-18T21:42:22.051 INFO:tasks.workunit.client.0.smithi099.stdout: "size": 3, 2024-08-18T21:42:22.051 INFO:tasks.workunit.client.0.smithi099.stdout: "min_size": 2, 2024-08-18T21:42:22.051 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_rule": 1, 2024-08-18T21:42:22.051 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-08-18T21:42:22.051 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-08-18T21:42:22.052 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-08-18T21:42:22.052 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-18T21:42:22.052 INFO:tasks.workunit.client.0.smithi099.stdout: "is_stretch_pool": false, 2024-08-18T21:42:22.052 INFO:tasks.workunit.client.0.smithi099.stdout: "object_hash": 2, 2024-08-18T21:42:22.052 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_autoscale_mode": "off", 2024-08-18T21:42:22.052 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num": 1, 2024-08-18T21:42:22.053 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num": 1, 2024-08-18T21:42:22.053 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num_target": 1, 2024-08-18T21:42:22.053 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_target": 1, 2024-08-18T21:42:22.053 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_pending": 1, 2024-08-18T21:42:22.053 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pg_merge_meta": { 2024-08-18T21:42:22.053 INFO:tasks.workunit.client.0.smithi099.stdout: "source_pgid": "0.0", 2024-08-18T21:42:22.054 INFO:tasks.workunit.client.0.smithi099.stdout: "ready_epoch": 0, 2024-08-18T21:42:22.054 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_started": 0, 2024-08-18T21:42:22.054 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": 0, 2024-08-18T21:42:22.054 INFO:tasks.workunit.client.0.smithi099.stdout: "source_version": "0'0", 2024-08-18T21:42:22.054 INFO:tasks.workunit.client.0.smithi099.stdout: "target_version": "0'0" 2024-08-18T21:42:22.055 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.055 INFO:tasks.workunit.client.0.smithi099.stdout: "last_change": "30", 2024-08-18T21:42:22.055 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend": "0", 2024-08-18T21:42:22.055 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-18T21:42:22.055 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-08-18T21:42:22.056 INFO:tasks.workunit.client.0.smithi099.stdout: "auid": 0, 2024-08-18T21:42:22.056 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_mode": "pool", 2024-08-18T21:42:22.056 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_seq": 1, 2024-08-18T21:42:22.056 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_epoch": 30, 2024-08-18T21:42:22.056 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_snaps": [ 2024-08-18T21:42:22.057 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.057 INFO:tasks.workunit.client.0.smithi099.stdout: "snapid": 1, 2024-08-18T21:42:22.057 INFO:tasks.workunit.client.0.smithi099.stdout: "stamp": "2024-08-18T21:41:11.330808+0000", 2024-08-18T21:42:22.057 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "snap1" 2024-08-18T21:42:22.057 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.057 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.058 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps": "[]", 2024-08-18T21:42:22.058 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_bytes": 0, 2024-08-18T21:42:22.058 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_objects": 0, 2024-08-18T21:42:22.058 INFO:tasks.workunit.client.0.smithi099.stdout: "tiers": [], 2024-08-18T21:42:22.058 INFO:tasks.workunit.client.0.smithi099.stdout: "tier_of": -1, 2024-08-18T21:42:22.059 INFO:tasks.workunit.client.0.smithi099.stdout: "read_tier": -1, 2024-08-18T21:42:22.059 INFO:tasks.workunit.client.0.smithi099.stdout: "write_tier": -1, 2024-08-18T21:42:22.059 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_mode": "none", 2024-08-18T21:42:22.059 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_bytes": 0, 2024-08-18T21:42:22.059 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_objects": 0, 2024-08-18T21:42:22.059 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-18T21:42:22.060 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-18T21:42:22.060 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-18T21:42:22.060 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_flush_age": 0, 2024-08-18T21:42:22.060 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_evict_age": 0, 2024-08-18T21:42:22.060 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profile": "myprofile", 2024-08-18T21:42:22.060 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_params": { 2024-08-18T21:42:22.061 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "none" 2024-08-18T21:42:22.061 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.061 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_period": 0, 2024-08-18T21:42:22.061 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_count": 0, 2024-08-18T21:42:22.061 INFO:tasks.workunit.client.0.smithi099.stdout: "use_gmt_hitset": true, 2024-08-18T21:42:22.062 INFO:tasks.workunit.client.0.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-08-18T21:42:22.062 INFO:tasks.workunit.client.0.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-08-18T21:42:22.062 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-08-18T21:42:22.062 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_search_last_n": 0, 2024-08-18T21:42:22.062 INFO:tasks.workunit.client.0.smithi099.stdout: "grade_table": [], 2024-08-18T21:42:22.063 INFO:tasks.workunit.client.0.smithi099.stdout: "stripe_width": 8192, 2024-08-18T21:42:22.063 INFO:tasks.workunit.client.0.smithi099.stdout: "expected_num_objects": 0, 2024-08-18T21:42:22.063 INFO:tasks.workunit.client.0.smithi099.stdout: "fast_read": false, 2024-08-18T21:42:22.063 INFO:tasks.workunit.client.0.smithi099.stdout: "options": {}, 2024-08-18T21:42:22.063 INFO:tasks.workunit.client.0.smithi099.stdout: "application_metadata": {} 2024-08-18T21:42:22.064 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.064 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.064 INFO:tasks.workunit.client.0.smithi099.stdout: "osds": [ 2024-08-18T21:42:22.064 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.064 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:42:22.064 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "b0d22869-f4e5-4905-ba11-85fef4467cd0", 2024-08-18T21:42:22.065 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:42:22.065 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:42:22.065 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:42:22.065 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:42:22.065 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 5, 2024-08-18T21:42:22.066 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 29, 2024-08-18T21:42:22.066 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 37, 2024-08-18T21:42:22.066 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 42, 2024-08-18T21:42:22.066 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 31, 2024-08-18T21:42:22.066 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:42:22.066 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:42:22.067 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.067 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.067 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.067 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6802", 2024-08-18T21:42:22.067 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 420852772 2024-08-18T21:42:22.068 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.068 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.068 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.068 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6803", 2024-08-18T21:42:22.068 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 420852772 2024-08-18T21:42:22.068 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.069 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.069 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.069 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:42:22.069 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.069 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.069 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.070 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6804", 2024-08-18T21:42:22.070 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 420852772 2024-08-18T21:42:22.070 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.070 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.070 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.071 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6805", 2024-08-18T21:42:22.071 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 420852772 2024-08-18T21:42:22.071 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.071 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.071 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.071 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:42:22.072 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.072 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.072 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.072 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6808", 2024-08-18T21:42:22.072 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 420852772 2024-08-18T21:42:22.073 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.073 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.073 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.073 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6809", 2024-08-18T21:42:22.073 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 420852772 2024-08-18T21:42:22.073 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.074 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.074 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.074 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:42:22.074 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.074 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.074 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.075 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6806", 2024-08-18T21:42:22.075 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 420852772 2024-08-18T21:42:22.075 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.075 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.075 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.075 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6807", 2024-08-18T21:42:22.075 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 420852772 2024-08-18T21:42:22.075 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.075 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.075 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.076 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6803/420852772", 2024-08-18T21:42:22.076 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6805/420852772", 2024-08-18T21:42:22.076 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6809/420852772", 2024-08-18T21:42:22.076 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6807/420852772", 2024-08-18T21:42:22.076 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:42:22.076 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:42:22.076 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:42:22.076 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.076 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.077 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.077 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:42:22.077 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "cf5d0225-bc36-4ab3-94ae-5d9e0c83d536", 2024-08-18T21:42:22.077 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 0, 2024-08-18T21:42:22.077 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:42:22.077 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:42:22.077 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:42:22.077 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 0, 2024-08-18T21:42:22.077 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 0, 2024-08-18T21:42:22.078 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 10, 2024-08-18T21:42:22.078 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 25, 2024-08-18T21:42:22.078 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 32, 2024-08-18T21:42:22.078 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:42:22.078 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:42:22.078 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.078 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.078 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.078 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:42:22.079 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:42:22.079 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.079 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.079 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.079 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:42:22.079 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:42:22.079 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.079 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.079 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.079 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:42:22.080 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.080 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.080 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.080 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:42:22.080 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:42:22.080 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.080 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.080 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.080 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:42:22.081 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:42:22.081 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.081 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.081 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.081 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:42:22.081 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.081 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.081 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.081 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:42:22.082 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:42:22.082 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.082 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.082 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.082 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:42:22.082 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:42:22.082 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.082 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.083 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.083 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:42:22.083 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.083 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.083 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.083 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:42:22.083 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:42:22.083 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.083 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.083 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.084 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:42:22.084 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:42:22.084 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.084 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.084 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.084 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/2915049588", 2024-08-18T21:42:22.084 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/2915049588", 2024-08-18T21:42:22.084 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2915049588", 2024-08-18T21:42:22.085 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2915049588", 2024-08-18T21:42:22.085 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:42:22.085 INFO:tasks.workunit.client.0.smithi099.stdout: "exists" 2024-08-18T21:42:22.085 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.085 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.085 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.085 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:42:22.085 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "a517c788-2442-48b9-984a-35f904f61dbf", 2024-08-18T21:42:22.085 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:42:22.086 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:42:22.086 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:42:22.086 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:42:22.086 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 15, 2024-08-18T21:42:22.086 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 29, 2024-08-18T21:42:22.086 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 39, 2024-08-18T21:42:22.087 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 0, 2024-08-18T21:42:22.087 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 33, 2024-08-18T21:42:22.087 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:42:22.087 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:42:22.087 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.087 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.088 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.088 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:42:22.088 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2898902377 2024-08-18T21:42:22.088 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.088 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.088 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.089 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:42:22.089 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2898902377 2024-08-18T21:42:22.089 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.089 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.089 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.089 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:42:22.089 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.090 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.090 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.090 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:42:22.090 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2898902377 2024-08-18T21:42:22.090 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.090 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.091 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.091 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:42:22.091 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2898902377 2024-08-18T21:42:22.091 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.091 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.092 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.092 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:42:22.092 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.092 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.092 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.092 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:42:22.093 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2898902377 2024-08-18T21:42:22.093 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.093 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.093 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.093 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:42:22.094 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2898902377 2024-08-18T21:42:22.094 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.094 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.094 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.094 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:42:22.094 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:42:22.095 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.095 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:42:22.095 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:42:22.095 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2898902377 2024-08-18T21:42:22.095 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.095 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.096 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:42:22.096 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:42:22.096 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2898902377 2024-08-18T21:42:22.096 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.096 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.096 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.097 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/2898902377", 2024-08-18T21:42:22.097 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/2898902377", 2024-08-18T21:42:22.097 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2898902377", 2024-08-18T21:42:22.097 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2898902377", 2024-08-18T21:42:22.097 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:42:22.098 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:42:22.098 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:42:22.098 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.098 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.098 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_xinfo": [ 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:41:22.483700+0000", 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:38:59.368785+0000", 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.099 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:41:23.146030+0000", 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:39:38.019442+0000", 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:41:23.785060+0000", 2024-08-18T21:42:22.100 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:40:16.092840+0000", 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap": [], 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_items": [], 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_primaries": [], 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_temp": [], 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_temp": [], 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "blocklist": {}, 2024-08-18T21:42:22.101 INFO:tasks.workunit.client.0.smithi099.stdout: "range_blocklist": {}, 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profiles": { 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "default": { 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "k": "2", 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "m": "2", 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "plugin": "jerasure", 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "technique": "reed_sol_van" 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "myprofile": { 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "crush-device-class": "", 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "crush-failure-domain": "osd", 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "crush-num-failure-domains": "0", 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "crush-osds-per-failure-domain": "0", 2024-08-18T21:42:22.102 INFO:tasks.workunit.client.0.smithi099.stdout: "crush-root": "default", 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: "jerasure-per-chunk-alignment": "false", 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: "k": "2", 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: "m": "1", 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: "plugin": "jerasure", 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: "technique": "reed_sol_van", 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: "w": "8" 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps_queue": [], 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: "new_removed_snaps": [], 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: "new_purged_snaps": [], 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_node_flags": {}, 2024-08-18T21:42:22.103 INFO:tasks.workunit.client.0.smithi099.stdout: "device_class_flags": {}, 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": { 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_enabled": false, 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_bucket_count": 0, 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: "degraded_stretch_mode": 0, 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: "recovering_stretch_mode": 0, 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_bucket": 0 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_metadata": [ 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:42:22.104 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6804/420852772,v1:127.0.0.1:6805/420852772]", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-18T21:42:22.105 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:38:29.376561Z", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:42:22.106 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6802/420852772,v1:127.0.0.1:6803/420852772]", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/420852772,v1:127.0.0.1:6809/420852772]", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/420852772,v1:127.0.0.1:6807/420852772]", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:42:22.107 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/0", 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:42:22.108 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/2915049588,v1:127.0.0.1:6813/2915049588]", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:42:22.109 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:39:07.375934Z", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:42:22.110 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/2915049588,v1:127.0.0.1:6811/2915049588]", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2915049588,v1:127.0.0.1:6817/2915049588]", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2915049588,v1:127.0.0.1:6815/2915049588]", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:42:22.111 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/1", 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/2898902377,v1:127.0.0.1:6813/2898902377]", 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:42:22.112 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:42:22.113 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:39:45.474255Z", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:42:22.114 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/2898902377,v1:127.0.0.1:6811/2898902377]", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2898902377,v1:127.0.0.1:6817/2898902377]", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2898902377,v1:127.0.0.1:6815/2898902377]", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:42:22.115 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/2", 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_clean_epochs": { 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: "min_last_epoch_clean": 0, 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": { 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: "per_pool": [] 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.116 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_epochs": [ 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 37 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 39 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_first_committed": 1, 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_last_committed": 44, 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: "crushmap": { 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": [ 2024-08-18T21:42:22.117 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.0", 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.1", 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.2", 2024-08-18T21:42:22.118 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:42:22.119 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.119 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.119 INFO:tasks.workunit.client.0.smithi099.stdout: "types": [ 2024-08-18T21:42:22.119 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.122 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 0, 2024-08-18T21:42:22.122 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd" 2024-08-18T21:42:22.122 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.122 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.122 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:42:22.122 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "host" 2024-08-18T21:42:22.122 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.122 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.122 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 2, 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "chassis" 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 3, 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "rack" 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 4, 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "row" 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 5, 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pdu" 2024-08-18T21:42:22.123 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 6, 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pod" 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 7, 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "room" 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 8, 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "datacenter" 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.124 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 9, 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "zone" 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 10, 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "region" 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "root" 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: "buckets": [ 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.125 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -1, 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default", 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.126 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -2, 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default~hdd", 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:42:22.127 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099", 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.128 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:42:22.129 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099~hdd", 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:42:22.130 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.131 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: "rules": [ 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_id": 0, 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_name": "replicated_rule", 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: "steps": [ 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "take", 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: "item": -1, 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: "item_name": "default" 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.132 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "choose_firstn", 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 0, 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "osd" 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "emit" 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_id": 1, 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_name": "test", 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 3, 2024-08-18T21:42:22.133 INFO:tasks.workunit.client.0.smithi099.stdout: "steps": [ 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "set_chooseleaf_tries", 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 5 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "set_choose_tries", 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 100 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "take", 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: "item": -1, 2024-08-18T21:42:22.134 INFO:tasks.workunit.client.0.smithi099.stdout: "item_name": "default" 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "choose_indep", 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 0, 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "osd" 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "emit" 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.136 INFO:tasks.workunit.client.0.smithi099.stderr:report 1895114369 2024-08-18T21:42:22.136 INFO:tasks.workunit.client.0.smithi099.stdout: "tunables": { 2024-08-18T21:42:22.136 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_tries": 0, 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_fallback_tries": 0, 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_total_tries": 50, 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_descend_once": 1, 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_vary_r": 1, 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_stable": 1, 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_descents": 100, 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_collision_tries": 100, 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "straw_calc_version": 1, 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "allowed_bucket_algs": 54, 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "profile": "jewel", 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_tunables": 1, 2024-08-18T21:42:22.137 INFO:tasks.workunit.client.0.smithi099.stdout: "legacy_tunables": 0, 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: "minimum_required_version": "jewel", 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables": 1, 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables2": 1, 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v2_rules": 1, 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables3": 1, 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v3_rules": 0, 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v4_buckets": 1, 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables5": 1, 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v5_rules": 0, 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: "has_msr_rules": 0 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.138 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_args": {} 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "fsmap": { 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "btime": "2024-08-18T21:38:22:260557+0000", 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "default_fscid": -1, 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": { 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": {}, 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "ro_compat": {}, 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "incompat": { 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_1": "base v0.20", 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_2": "client writeable ranges", 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_3": "default file layouts on dirs", 2024-08-18T21:42:22.139 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_4": "dir inode in separate object", 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_5": "mds uses versioned encoding", 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_8": "no anchor table", 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_9": "file layout v2", 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_10": "snaprealm v2", 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_11": "minor log segments", 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_12": "quiesce subvolumes" 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_flags": { 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: "enable_multiple": true, 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: "ever_enabled_multiple": true 2024-08-18T21:42:22.140 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "standbys": [], 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "filesystems": [] 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_first_committed": 1, 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_last_committed": 1, 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "auth": { 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 5, 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "num_secrets": 3 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 1, 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_active": 1, 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_unknown": 0, 2024-08-18T21:42:22.141 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osd": 3, 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_sum": { 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 62667, 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 101, 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 1, 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 303, 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 101, 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 101, 2024-08-18T21:42:22.142 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 101, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 101, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:42:22.143 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:42:22.144 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 0, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 0, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:42:22.145 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 18, 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 18, 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 3, 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 3, 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 0 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum": { 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 3, 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:42:22.146 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83708, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2180, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314489092, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036830208, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2232320, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 1556993, 2024-08-18T21:42:22.147 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10396, 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416932 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.148 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum_by_class": { 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: "hdd": { 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:42:22.149 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 3, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 83708, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2180, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 10, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 81461, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314489092, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322036830208, 2024-08-18T21:42:22.150 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2232320, 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 1556993, 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 10396, 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 83416932 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:42:22.151 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.152 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_stats": [ 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "poolid": 1, 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 1, 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 62667, 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 101, 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 1, 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 303, 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 101, 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:42:22.153 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 101, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 101, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 101, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:42:22.154 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:42:22.155 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 1241088, 2024-08-18T21:42:22.156 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 1241088, 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 18, 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 18, 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 3, 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 3, 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 3 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.157 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_stats": [ 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 158913789964 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 42949672997 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 167503724555 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.158 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_state": [ 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: "state": "active+undersized+degraded", 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 1 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_osd": [ 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2147483647, 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:42:22.159 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 1, 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.160 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 0, 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": [ 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": {} 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: "servicemap": { 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 5, 2024-08-18T21:42:22.161 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:41:17.893519+0000", 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: "services": {} 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_first_committed": 1, 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_last_committed": 108, 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_first_committed": 1, 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_last_committed": 429, 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: "paxos": { 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 586, 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pn": 0, 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: "accepted_pn": 0 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:42:22.162 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:42:22.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-18T21:42:22.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:42:22.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-08-18T21:42:22.164 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:42:22.164 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:42:22.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:42:22.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:42:22.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:42:22.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:42:22.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-08-18T21:42:22.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-18T21:42:22.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-18T21:42:22.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-08-18T21:42:22.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-18T21:42:22.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-08-18T21:42:22.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-08-18T21:42:23.822 INFO:tasks.workunit.client.0.smithi099.stderr:PG '1.0' not found 2024-08-18T21:42:24.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-08-18T21:42:24.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-08-18T21:42:24.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:42:24.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:42:24.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-18T21:42:24.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:42:24.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-08-18T21:42:24.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:42:24.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:42:24.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:42:24.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:42:24.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-08-18T21:42:24.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-08-18T21:42:24.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:42:24.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:42:24.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:42:24.368 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:42:24.368 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:42:24.368 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:42:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:42:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:42:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:42:24.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:42:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:42:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:42:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:42:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:42:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:42:24.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:42:24.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:42:24.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:42:24.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:42:24.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:42:24.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-08-18T21:42:24.371 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:42:24.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-18T21:42:24.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:42:24.372 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-08-18T21:42:24.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-18T21:42:24.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:42:24.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:42:24.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:42:24.413 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:42:24.416+0000 7f79c26bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:42:24.416 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:42:24.420+0000 7f79c26bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:42:24.417 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:42:24.420+0000 7f79c26bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:42:24.828 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-18T21:42:24.828 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:42:24.829 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:42:24.829 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:42:24.829 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:42:24.829 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:42:24.829 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:42:24.829 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:42:24.829 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:42:24.829 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:42:25.277 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 37 up_thru 42 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/420852772,v1:127.0.0.1:6803/420852772] [v2:127.0.0.1:6804/420852772,v1:127.0.0.1:6805/420852772] exists,up b0d22869-f4e5-4905-ba11-85fef4467cd0 2024-08-18T21:42:25.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:42:25.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:42:25.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:42:25.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:42:25.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-08-18T21:42:25.278 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:42:25.278 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:42:25.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:42:25.278 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:42:25.279 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:42:25.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:42:25.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-08-18T21:42:25.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-18T21:42:25.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-18T21:42:25.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-08-18T21:42:25.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-18T21:42:25.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-08-18T21:42:25.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-08-18T21:42:25.686 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:42:25.688+0000 7f79c26bb7c0 -1 Falling back to public interface 2024-08-18T21:42:27.074 INFO:tasks.workunit.client.0.smithi099.stderr:PG '1.0' not found 2024-08-18T21:42:27.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-08-18T21:42:27.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-08-18T21:42:27.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:42:27.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:42:27.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-18T21:42:27.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:42:27.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-08-18T21:42:27.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:42:27.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:42:27.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:42:27.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:42:27.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-08-18T21:42:27.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-08-18T21:42:27.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:42:27.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:42:27.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:42:27.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:42:27.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:42:27.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:42:27.372 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:42:27.372 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:42:27.372 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:42:27.372 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:42:27.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:42:27.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:42:27.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:42:27.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:42:27.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:42:27.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:42:27.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:42:27.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:42:27.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:42:27.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:42:27.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-08-18T21:42:27.373 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:42:27.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-18T21:42:27.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:42:27.374 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-08-18T21:42:27.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-18T21:42:27.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:42:27.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:42:27.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:42:27.416 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:42:27.420+0000 7f953f46a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:42:27.419 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:42:27.420+0000 7f953f46a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:42:27.420 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:42:27.424+0000 7f953f46a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:42:27.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-18T21:42:27.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:42:27.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:42:27.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:42:27.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:42:27.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:42:27.820 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:42:27.821 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:42:27.821 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:42:27.821 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:42:28.259 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/2898902377,v1:127.0.0.1:6811/2898902377] [v2:127.0.0.1:6812/2898902377,v1:127.0.0.1:6813/2898902377] exists,up a517c788-2442-48b9-984a-35f904f61dbf 2024-08-18T21:42:28.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:42:28.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:42:28.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:42:28.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-08-18T21:42:28.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-08-18T21:42:28.261 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-08-18T21:42:28.261 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-08-18T21:42:28.261 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-08-18T21:42:28.261 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-08-18T21:42:28.261 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-08-18T21:42:28.674 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:42:28.676+0000 7f953f46a7c0 -1 Falling back to public interface 2024-08-18T21:42:28.802 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:42:28.804+0000 7f79c26bb7c0 -1 osd.0 44 log_to_monitors true 2024-08-18T21:42:30.674 INFO:tasks.workunit.client.0.smithi099.stderr:PG '1.0' not found 2024-08-18T21:42:31.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-08-18T21:42:31.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-08-18T21:42:31.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-08-18T21:42:31.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:42:31.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:42:31.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:42:31.229 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-08-18T21:42:31.230 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:42:31.230 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:42:31.230 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:42:31.230 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:42:31.231 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:42:31.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-08-18T21:42:31.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:42:31.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:42:31.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:42:31.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:42:31.306 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:42:31.657 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:42:31.660+0000 7f953f46a7c0 -1 osd.2 44 log_to_monitors true 2024-08-18T21:42:31.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:42:31.753 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:42:31.754 INFO:tasks.workunit.client.0.smithi099.stderr:2' 2024-08-18T21:42:31.754 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:42:31.754 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:42:31.754 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:42:31.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=201863462915 2024-08-18T21:42:31.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 201863462915 2024-08-18T21:42:31.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-201863462915' 2024-08-18T21:42:31.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:42:31.918 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:42:32.073 INFO:tasks.workunit.client.0.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-18T21:42:32.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq= 2024-08-18T21:42:32.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z '' 2024-08-18T21:42:32.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: continue 2024-08-18T21:42:32.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:42:32.075 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:42:32.200 INFO:tasks.workunit.client.0.smithi099.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-08-18T21:42:32.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq= 2024-08-18T21:42:32.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z '' 2024-08-18T21:42:32.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: continue 2024-08-18T21:42:32.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:42:32.203 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-201863462915 2024-08-18T21:42:32.203 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:42:32.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:42:32.204 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-201863462915 2024-08-18T21:42:32.204 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:42:32.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=201863462915 2024-08-18T21:42:32.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 201863462915' 2024-08-18T21:42:32.206 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 201863462915 2024-08-18T21:42:32.206 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:42:32.664 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 201863462915 -lt 201863462915 2024-08-18T21:42:32.665 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:42:32.666 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:32.666 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:33.252 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:42:33.252 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:42:33.253 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:42:33.253 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:42:33.253 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:42:33.253 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:42:33.254 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:42:33.254 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:42:33.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:42:33.707 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:42:33.708 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:33.708 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:34.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:42:34.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-08-18T21:42:34.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-08-18T21:42:34.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-08-18T21:42:34.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:42:34.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-08-18T21:42:34.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:42:34.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:42:34.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:42:34.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:42:34.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:42:34.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:42:34.360 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:42:34.360 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:42:34.770 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:42:34.771 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:42:34.771 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:34.771 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:35.327 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:42:35.327 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:42:35.327 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:42:35.327 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:42:35.327 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:42:35.328 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:42:35.328 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:42:35.328 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:42:35.328 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:42:35.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:42:35.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:42:35.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-08-18T21:42:35.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:42:35.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-08-18T21:42:36.089 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:42:36.090 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:42:36.090 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:42:36.090 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:42:36.090 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:42:36.091 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:42:36.091 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:42:36.091 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:42:36.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:42:36.511 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:42:36.511 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:36.511 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:37.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:42:37.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:42:37.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:42:37.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:42:37.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:42:37.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:42:37.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:42:37.061 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:42:37.061 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:42:37.615 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:42:37.615 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:42:37.615 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-08-18T21:42:37.615 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:42:37.615 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-08-18T21:42:38.016 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:42:38.016 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:42:38.017 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:42:38.017 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:42:38.017 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:42:38.017 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:42:38.018 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:42:38.018 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:42:38.461 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:42:38.462 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:42:38.462 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:38.462 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:39.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:42:39.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:42:39.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:42:39.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:42:39.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:42:39.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:42:39.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:42:39.009 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:42:39.009 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:42:39.598 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:42:39.598 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:42:39.598 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-08-18T21:42:39.599 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:42:39.599 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-08-18T21:42:40.399 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:42:40.400 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:42:40.400 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:42:40.401 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:42:40.401 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:42:40.401 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:42:40.401 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:42:40.401 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:42:40.830 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:42:40.831 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:42:40.831 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:40.831 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:41.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:42:41.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:42:41.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:42:41.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:42:41.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:42:41.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:42:41.386 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:42:41.386 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:42:41.386 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:42:41.951 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:42:41.952 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:42:41.952 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-08-18T21:42:41.952 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:42:41.952 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-08-18T21:42:43.553 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:42:43.553 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:42:43.554 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:42:43.554 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:42:43.554 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:42:43.554 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:42:43.555 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:42:43.555 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:42:43.992 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:42:43.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:42:43.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:43.992 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:44.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:42:44.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:42:44.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:42:44.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:42:44.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:42:44.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:42:44.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:42:44.590 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:42:44.590 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:42:45.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:42:45.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:42:45.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-08-18T21:42:45.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:42:45.169 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-08-18T21:42:48.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:42:48.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:42:48.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:42:48.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:42:48.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:42:48.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:42:48.372 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:42:48.372 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:42:48.791 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:42:48.792 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:42:48.792 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:48.792 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:49.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:42:49.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:42:49.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:42:49.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:42:49.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:42:49.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:42:49.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:42:49.358 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:42:49.358 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:42:49.897 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:42:49.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:42:49.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-08-18T21:42:49.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:42:49.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-08-18T21:42:56.299 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:42:56.299 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:42:56.300 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:42:56.300 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:42:56.300 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:42:56.300 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:42:56.301 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:42:56.301 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:42:56.742 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:42:56.743 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:42:56.743 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:42:56.743 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:42:57.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:42:57.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:42:57.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:42:57.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:42:57.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:42:57.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:42:57.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:42:57.334 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:42:57.334 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:42:57.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:42:57.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:42:57.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-08-18T21:42:57.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-08-18T21:42:57.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-08-18T21:43:05.237 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-08-18T21:43:05.238 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:43:05.238 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:43:05.238 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:43:05.238 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:43:05.239 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:43:05.239 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:43:05.239 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:43:05.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-08-18T21:43:05.688 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:43:05.688 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:43:05.688 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:43:06.251 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-08-18T21:43:06.251 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-08-18T21:43:06.251 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-08-18T21:43:06.251 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-08-18T21:43:06.251 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-08-18T21:43:06.251 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-08-18T21:43:06.252 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-08-18T21:43:06.252 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-08-18T21:43:06.252 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-08-18T21:43:06.811 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-08-18T21:43:06.811 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-08-18T21:43:06.811 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-08-18T21:43:06.811 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-08-18T21:43:07.370 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:43:07.370 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_fingerprint": "1749aa68-d2b4-498a-a3c5-2ae6c810744e", 2024-08-18T21:43:07.370 INFO:tasks.workunit.client.0.smithi099.stdout: "version": "19.0.0-5847-g79766939", 2024-08-18T21:43:07.370 INFO:tasks.workunit.client.0.smithi099.stdout: "commit": "797669399b32d2915e64eb6d4b1257ae3c50f80b", 2024-08-18T21:43:07.370 INFO:tasks.workunit.client.0.smithi099.stdout: "timestamp": "2024-08-18T21:43:07.374676+0000", 2024-08-18T21:43:07.370 INFO:tasks.workunit.client.0.smithi099.stdout: "tag": "", 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: "health": { 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: "status": "HEALTH_WARN", 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: "checks": { 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: "OSDMAP_FLAGS": { 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "nodown,noout,noin flag(s) set", 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 17 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [], 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:43:07.371 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: "OSD_DOWN": { 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 osds down", 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "osd.1 (root=default,host=smithi099) is down" 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.372 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: "PG_DEGRADED": { 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.373 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: "POOL_APP_NOT_ENABLED": { 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: "severity": "HEALTH_WARN", 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: "summary": { 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "1 pool(s) do not have an application enabled", 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: "count": 1 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: "detail": [ 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "application not enabled on pool 'test'" 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.374 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: "muted": false 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: "mutes": [] 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_first_committed": 1, 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap_last_committed": 1, 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: "monmap": { 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:43:07.375 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:38:21.266415Z", 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:38:21.266415Z", 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release": 19, 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "min_mon_release_name": "squid", 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "election_strategy": 1, 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "disallowed_leaders: ": "", 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": false, 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "tiebreaker_mon": "", 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_ranks: ": "", 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "features": { 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "persistent": [ 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "kraken", 2024-08-18T21:43:07.376 INFO:tasks.workunit.client.0.smithi099.stdout: "luminous", 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: "mimic", 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap-prune", 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: "nautilus", 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: "octopus", 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: "pacific", 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: "elector-pinging", 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: "quincy", 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: "reef", 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: "squid" 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: "optional": [] 2024-08-18T21:43:07.377 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.378 INFO:tasks.workunit.client.0.smithi099.stdout: "mons": [ 2024-08-18T21:43:07.378 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.378 INFO:tasks.workunit.client.0.smithi099.stdout: "rank": 0, 2024-08-18T21:43:07.378 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "a", 2024-08-18T21:43:07.378 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:43:07.378 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.378 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.378 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.378 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115", 2024-08-18T21:43:07.381 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 0 2024-08-18T21:43:07.381 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.381 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:7115/0", 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:7115/0", 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: "priority": 0, 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 0, 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_location": "{}" 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: "quorum": [ 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: 0 2024-08-18T21:43:07.382 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap": { 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 53, 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "fsid": "d6dec24a-9728-40c3-9343-6a2e4528b186", 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "created": "2024-08-18T21:38:22.293071+0000", 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:42:32.811368+0000", 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "last_up_change": "2024-08-18T21:42:32.208071+0000", 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "last_in_change": "2024-08-18T21:41:37.569144+0000", 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_num": 5800384, 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_set": [ 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "nodown", 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "noin", 2024-08-18T21:43:07.383 INFO:tasks.workunit.client.0.smithi099.stdout: "noout", 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: "pglog_hardlimit", 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snapdirs", 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: "recovery_deletes", 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: "sortbitwise" 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_version": 12, 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: "full_ratio": 0.99000000953674316, 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: "nearfull_ratio": 0.85000002384185791, 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_snapshot": "", 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_max": 1, 2024-08-18T21:43:07.384 INFO:tasks.workunit.client.0.smithi099.stdout: "max_osd": 3, 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "require_min_compat_client": "luminous", 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "min_compat_client": "jewel", 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "require_osd_release": "squid", 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "allow_crimson": false, 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "pools": [ 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_name": "test", 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "create_time": "2024-08-18T21:40:21.104981+0000", 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "flags": 16389, 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 3, 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "size": 3, 2024-08-18T21:43:07.385 INFO:tasks.workunit.client.0.smithi099.stdout: "min_size": 2, 2024-08-18T21:43:07.386 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_rule": 1, 2024-08-18T21:43:07.386 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_count": 0, 2024-08-18T21:43:07.386 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_target": 0, 2024-08-18T21:43:07.386 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_barrier": 0, 2024-08-18T21:43:07.386 INFO:tasks.workunit.client.0.smithi099.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-08-18T21:43:07.386 INFO:tasks.workunit.client.0.smithi099.stdout: "is_stretch_pool": false, 2024-08-18T21:43:07.386 INFO:tasks.workunit.client.0.smithi099.stdout: "object_hash": 2, 2024-08-18T21:43:07.386 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_autoscale_mode": "off", 2024-08-18T21:43:07.386 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num": 1, 2024-08-18T21:43:07.386 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num": 1, 2024-08-18T21:43:07.386 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_placement_num_target": 1, 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_target": 1, 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_num_pending": 1, 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pg_merge_meta": { 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: "source_pgid": "0.0", 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: "ready_epoch": 0, 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_started": 0, 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": 0, 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: "source_version": "0'0", 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: "target_version": "0'0" 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: "last_change": "30", 2024-08-18T21:43:07.387 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend": "0", 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_prenautilus": "0", 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: "last_force_op_resend_preluminous": "0", 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: "auid": 0, 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_mode": "pool", 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_seq": 1, 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_epoch": 30, 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_snaps": [ 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: "snapid": 1, 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: "stamp": "2024-08-18T21:41:11.330808+0000", 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "snap1" 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.388 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps": "[]", 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_bytes": 0, 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "quota_max_objects": 0, 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "tiers": [], 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "tier_of": -1, 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "read_tier": -1, 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "write_tier": -1, 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_mode": "none", 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_bytes": 0, 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "target_max_objects": 0, 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_target_full_ratio_micro": 800000, 2024-08-18T21:43:07.389 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_flush_age": 0, 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: "cache_min_evict_age": 0, 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profile": "myprofile", 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_params": { 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "none" 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_period": 0, 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_count": 0, 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: "use_gmt_hitset": true, 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: "min_read_recency_for_promote": 0, 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: "min_write_recency_for_promote": 0, 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_grade_decay_rate": 0, 2024-08-18T21:43:07.390 INFO:tasks.workunit.client.0.smithi099.stdout: "hit_set_search_last_n": 0, 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: "grade_table": [], 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: "stripe_width": 8192, 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: "expected_num_objects": 0, 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: "fast_read": false, 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: "options": {}, 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: "application_metadata": {} 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: "osds": [ 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "b0d22869-f4e5-4905-ba11-85fef4467cd0", 2024-08-18T21:43:07.391 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 37, 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 44, 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 47, 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 51, 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 46, 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.392 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6802", 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1388599734 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6803", 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1388599734 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:43:07.393 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6804", 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1388599734 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6805", 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1388599734 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.394 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6808", 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1388599734 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6809", 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1388599734 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.395 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6806", 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1388599734 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6807", 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1388599734 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.396 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6803/1388599734", 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6805/1388599734", 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1388599734", 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1388599734", 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "cf5d0225-bc36-4ab3-94ae-5d9e0c83d536", 2024-08-18T21:43:07.397 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 0, 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 0, 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 0, 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 10, 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 25, 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 32, 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.398 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.399 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:43:07.400 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:43:07.401 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 2915049588 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.402 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/2915049588", 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/2915049588", 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2915049588", 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2915049588", 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: "exists" 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: "uuid": "a517c788-2442-48b9-984a-35f904f61dbf", 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 1, 2024-08-18T21:43:07.403 INFO:tasks.workunit.client.0.smithi099.stdout: "in": 1, 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 1, 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_affinity": 1, 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_begin": 39, 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "last_clean_end": 44, 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 51, 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "up_thru": 0, 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "down_at": 50, 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "lost_at": 0, 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addrs": { 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6810", 2024-08-18T21:43:07.404 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1576157991 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6811", 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1576157991 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addrs": { 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6812", 2024-08-18T21:43:07.405 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1576157991 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6813", 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1576157991 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addrs": { 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.406 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6816", 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1576157991 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6817", 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1576157991 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addrs": { 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: "addrvec": [ 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.407 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v2", 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6814", 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1576157991 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "v1", 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: "addr": "127.0.0.1:6815", 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: "nonce": 1576157991 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: "public_addr": "127.0.0.1:6811/1576157991", 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: "cluster_addr": "127.0.0.1:6813/1576157991", 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1576157991", 2024-08-18T21:43:07.408 INFO:tasks.workunit.client.0.smithi099.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1576157991", 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: "state": [ 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: "exists", 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: "up" 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_xinfo": [ 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:42:29.114132+0000", 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:43:07.409 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:38:59.368785+0000", 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:41:23.146030+0000", 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:39:38.019442+0000", 2024-08-18T21:43:07.410 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: "down_stamp": "2024-08-18T21:42:31.998874+0000", 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_probability": 0, 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: "laggy_interval": 0, 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: "features": 4540701547738038271, 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: "old_weight": 0, 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: "last_purged_snaps_scrub": "2024-08-18T21:40:16.092840+0000", 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: "dead_epoch": 0 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap": [], 2024-08-18T21:43:07.411 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_items": [], 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_upmap_primaries": [], 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "pg_temp": [], 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "primary_temp": [], 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "blocklist": {}, 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "range_blocklist": {}, 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "erasure_code_profiles": { 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "default": { 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "k": "2", 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "m": "2", 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "plugin": "jerasure", 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "technique": "reed_sol_van" 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.412 INFO:tasks.workunit.client.0.smithi099.stdout: "myprofile": { 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "crush-device-class": "", 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "crush-failure-domain": "osd", 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "crush-num-failure-domains": "0", 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "crush-osds-per-failure-domain": "0", 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "crush-root": "default", 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "jerasure-per-chunk-alignment": "false", 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "k": "2", 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "m": "1", 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "plugin": "jerasure", 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "technique": "reed_sol_van", 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "w": "8" 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.413 INFO:tasks.workunit.client.0.smithi099.stdout: "removed_snaps_queue": [], 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: "new_removed_snaps": [], 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: "new_purged_snaps": [], 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: "crush_node_flags": {}, 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: "device_class_flags": {}, 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode": { 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_enabled": false, 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_bucket_count": 0, 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: "degraded_stretch_mode": 0, 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: "recovering_stretch_mode": 0, 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: "stretch_mode_bucket": 0 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.414 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_metadata": [ 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6804/1388599734,v1:127.0.0.1:6805/1388599734]", 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:43:07.415 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:43:07.416 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:38:29.376561Z", 2024-08-18T21:43:07.417 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:43:07.417 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:43:07.417 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:43:07.417 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:43:07.417 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:43:07.417 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:43:07.417 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:43:07.417 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6802/1388599734,v1:127.0.0.1:6803/1388599734]", 2024-08-18T21:43:07.417 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1388599734,v1:127.0.0.1:6809/1388599734]", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1388599734,v1:127.0.0.1:6807/1388599734]", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/0", 2024-08-18T21:43:07.418 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/2915049588,v1:127.0.0.1:6813/2915049588]", 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:43:07.419 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:43:07.420 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:39:07.375934Z", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/2915049588,v1:127.0.0.1:6811/2915049588]", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:43:07.421 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2915049588,v1:127.0.0.1:6817/2915049588]", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2915049588,v1:127.0.0.1:6815/2915049588]", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/1", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:43:07.422 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: "arch": "x86_64", 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: "back_addr": "[v2:127.0.0.1:6812/1576157991,v1:127.0.0.1:6813/1576157991]", 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: "back_iface": "", 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs": "1", 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_db": "0", 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_dedicated_wal": "0", 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: "bluefs_single_shared_device": "1", 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_allocation_from_file": "0", 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_access_mode": "file", 2024-08-18T21:43:07.423 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_block_size": "4096", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_devices": "sda", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_rotational": "1", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_size": "107374182400", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_support_discard": "0", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_bdev_type": "hdd", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "bluestore_min_alloc_size": "4096", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_release": "squid", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_short": "19.0.0-5847-g79766939", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "ceph_version_when_created": "ceph version 19.0.0-5847-g79766939 (797669399b32d2915e64eb6d4b1257ae3c50f80b) squid (dev)", 2024-08-18T21:43:07.424 INFO:tasks.workunit.client.0.smithi099.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "created_at": "2024-08-18T21:39:45.474255Z", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "default_device_class": "hdd", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5LHPN", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": "sda", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "distro": "ubuntu", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "distro_version": "22.04", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "front_addr": "[v2:127.0.0.1:6810/1576157991,v1:127.0.0.1:6811/1576157991]", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "front_iface": "", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1576157991,v1:127.0.0.1:6817/1576157991]", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1576157991,v1:127.0.0.1:6815/1576157991]", 2024-08-18T21:43:07.425 INFO:tasks.workunit.client.0.smithi099.stdout: "hostname": "smithi099", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "journal_rotational": "1", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_description": "#128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "kernel_version": "5.15.0-118-generic", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_swap_kb": "2097148", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "mem_total_kb": "32746700", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "objectstore_numa_unknown_devices": "sda", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "os": "Linux", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_data": "td/divergent-priors/2", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_objectstore": "bluestore", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "osdspec_affinity": "", 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: "rotational": "1" 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_clean_epochs": { 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: "min_last_epoch_clean": 26, 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: "last_epoch_clean": { 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: "per_pool": [ 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: "poolid": 1, 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: "floor": 26 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_epochs": [ 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.427 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 48 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 51 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_first_committed": 1, 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: "osdmap_last_committed": 53, 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: "crushmap": { 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: "devices": [ 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.428 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.0", 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.1", 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd.2", 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: "class": "hdd" 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.429 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: "types": [ 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 0, 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "osd" 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "host" 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 2, 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "chassis" 2024-08-18T21:43:07.430 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 3, 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "rack" 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 4, 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "row" 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 5, 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pdu" 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.431 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 6, 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "pod" 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 7, 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "room" 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 8, 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "datacenter" 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.432 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 9, 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "zone" 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 10, 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "region" 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "root" 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.433 INFO:tasks.workunit.client.0.smithi099.stdout: "buckets": [ 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -1, 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default", 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:43:07.434 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -2, 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "default~hdd", 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 11, 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "root", 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:43:07.435 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -3, 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099", 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:43:07.436 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.437 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: "id": -4, 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: "name": "smithi099~hdd", 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: "type_id": 1, 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: "type_name": "host", 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 19206, 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: "alg": "straw2", 2024-08-18T21:43:07.438 INFO:tasks.workunit.client.0.smithi099.stdout: "hash": "rjenkins1", 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: "items": [ 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 0, 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 0 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 1, 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 1 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: "id": 2, 2024-08-18T21:43:07.439 INFO:tasks.workunit.client.0.smithi099.stdout: "weight": 6402, 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: "pos": 2 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: "rules": [ 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_id": 0, 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_name": "replicated_rule", 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 1, 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: "steps": [ 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "take", 2024-08-18T21:43:07.440 INFO:tasks.workunit.client.0.smithi099.stdout: "item": -1, 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: "item_name": "default" 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "choose_firstn", 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 0, 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "osd" 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "emit" 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.441 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_id": 1, 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: "rule_name": "test", 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: "type": 3, 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: "steps": [ 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "set_chooseleaf_tries", 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 5 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "set_choose_tries", 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 100 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.442 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "take", 2024-08-18T21:43:07.443 INFO:tasks.workunit.client.0.smithi099.stdout: "item": -1, 2024-08-18T21:43:07.443 INFO:tasks.workunit.client.0.smithi099.stdout: "item_name": "default" 2024-08-18T21:43:07.443 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.443 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.443 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "choose_indep", 2024-08-18T21:43:07.443 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 0, 2024-08-18T21:43:07.443 INFO:tasks.workunit.client.0.smithi099.stdout: "type": "osd" 2024-08-18T21:43:07.443 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.444 INFO:tasks.workunit.client.0.smithi099.stderr:report 4061514744 2024-08-18T21:43:07.444 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.444 INFO:tasks.workunit.client.0.smithi099.stdout: "op": "emit" 2024-08-18T21:43:07.444 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.444 INFO:tasks.workunit.client.0.smithi099.stdout: ] 2024-08-18T21:43:07.444 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.444 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.444 INFO:tasks.workunit.client.0.smithi099.stdout: "tunables": { 2024-08-18T21:43:07.444 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_tries": 0, 2024-08-18T21:43:07.444 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_local_fallback_tries": 0, 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_total_tries": 50, 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_descend_once": 1, 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_vary_r": 1, 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "chooseleaf_stable": 1, 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_descents": 100, 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "msr_collision_tries": 100, 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "straw_calc_version": 1, 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "allowed_bucket_algs": 54, 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "profile": "jewel", 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "optimal_tunables": 1, 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "legacy_tunables": 0, 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "minimum_required_version": "jewel", 2024-08-18T21:43:07.445 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables": 1, 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables2": 1, 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v2_rules": 1, 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables3": 1, 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v3_rules": 0, 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v4_buckets": 1, 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "require_feature_tunables5": 1, 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "has_v5_rules": 0, 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "has_msr_rules": 0 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "choose_args": {} 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "fsmap": { 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 1, 2024-08-18T21:43:07.446 INFO:tasks.workunit.client.0.smithi099.stdout: "btime": "2024-08-18T21:38:22:260557+0000", 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "default_fscid": -1, 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": { 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "compat": {}, 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "ro_compat": {}, 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "incompat": { 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_1": "base v0.20", 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_2": "client writeable ranges", 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_3": "default file layouts on dirs", 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_4": "dir inode in separate object", 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_5": "mds uses versioned encoding", 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_6": "dirfrag is stored in omap", 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_8": "no anchor table", 2024-08-18T21:43:07.447 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_9": "file layout v2", 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_10": "snaprealm v2", 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_11": "minor log segments", 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_12": "quiesce subvolumes" 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: "feature_flags": { 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: "enable_multiple": true, 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: "ever_enabled_multiple": true 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: "standbys": [], 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: "filesystems": [] 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_first_committed": 1, 2024-08-18T21:43:07.448 INFO:tasks.workunit.client.0.smithi099.stdout: "mdsmap_last_committed": 1, 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "auth": { 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 5, 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "num_secrets": 3 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 1, 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_active": 1, 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_unknown": 0, 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osd": 3, 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_sum": { 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 62667, 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 101, 2024-08-18T21:43:07.449 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 1, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 303, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 101, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 101, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 101, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 101, 2024-08-18T21:43:07.450 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:43:07.451 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:43:07.452 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 0, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 0, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 18, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 18, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 3, 2024-08-18T21:43:07.453 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 3, 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 0 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum": { 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 3, 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 85160, 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2324, 2024-08-18T21:43:07.454 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 15, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 82736, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314487640, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322035343360, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2379776, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 1618177, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 16263, 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 84721785 2024-08-18T21:43:07.455 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:43:07.456 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_sum_by_class": { 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "hdd": { 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "up_from": 0, 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 0, 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pgs": 3, 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "num_osds": 3, 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_osds": 3, 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "num_per_pool_omap_osds": 3, 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "kb": 314572800, 2024-08-18T21:43:07.457 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used": 85160, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_data": 2324, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_omap": 15, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_used_meta": 82736, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "kb_avail": 314487640, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "statfs": { 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 322122547200, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 322035343360, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 2379776, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 1618177, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:43:07.458 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 16263, 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 84721785 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: "hb_peers": [], 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: "snap_trim_queue_len": 0, 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: "num_snap_trimming": 0, 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shards_repaired": 0, 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: "op_queue_age_hist": { 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: "histogram": [], 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: "upper_bound": 1 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: "perf_stat": { 2024-08-18T21:43:07.459 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ms": 0, 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ms": 0, 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: "commit_latency_ns": 0, 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: "apply_latency_ns": 0 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: "alerts": [], 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: "network_ping_times": [] 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: "pool_stats": [ 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: "poolid": 1, 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg": 1, 2024-08-18T21:43:07.460 INFO:tasks.workunit.client.0.smithi099.stdout: "stat_sum": { 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes": 62667, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects": 101, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_clones": 1, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_object_copies": 303, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing_on_primary": 0, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_missing": 0, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_degraded": 101, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_misplaced": 0, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_unfound": 0, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_dirty": 101, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_whiteouts": 0, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read": 0, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_read_kb": 0, 2024-08-18T21:43:07.461 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write": 101, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_write_kb": 101, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_scrub_errors": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_shallow_scrub_errors": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_deep_scrub_errors": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_recovered": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_recovered": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_keys_recovered": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_omap": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_hit_set_archive": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_bytes_hit_set_archive": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_kb": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict": 0, 2024-08-18T21:43:07.462 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_kb": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_promote": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_high": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_flush_mode_low": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_some": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_evict_mode_full": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_pinned": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_legacy_snapsets": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_large_omap_objects": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_manifest": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_bytes": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_omap_keys": 0, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "num_objects_repaired": 0 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.463 INFO:tasks.workunit.client.0.smithi099.stdout: "store_stats": { 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "total": 0, 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "available": 0, 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "internally_reserved": 0, 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "allocated": 1241088, 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "data_stored": 1241088, 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed": 0, 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_allocated": 0, 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "data_compressed_original": 0, 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "omap_allocated": 0, 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "internal_metadata": 0 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "log_size": 18, 2024-08-18T21:43:07.464 INFO:tasks.workunit.client.0.smithi099.stdout: "ondisk_log_size": 18, 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: "up": 3, 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: "acting": 3, 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: "num_store_stats": 3 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: "osd_stats": [ 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 201863462922 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:43:07.465 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 42949672997 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: "seq": 219043332103 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_state": [ 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: "state": "active+undersized+degraded", 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: "num": 1 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.466 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: "num_pg_by_osd": [ 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2147483647, 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 2, 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.467 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 0, 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 1, 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 1, 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: "osd": 1, 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: "num_primary_pg": 0, 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: "num_acting_pg": 0, 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: "num_up_not_acting_pg": 0 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": [ 2024-08-18T21:43:07.468 INFO:tasks.workunit.client.0.smithi099.stdout: { 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: "pool": 1, 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: "purged_snaps": {} 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: ], 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: "servicemap": { 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: "epoch": 5, 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: "modified": "2024-08-18T21:41:17.893519+0000", 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: "services": {} 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: }, 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_first_committed": 1, 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: "mgrstat_last_committed": 131, 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_first_committed": 1, 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: "logm_last_committed": 508, 2024-08-18T21:43:07.469 INFO:tasks.workunit.client.0.smithi099.stdout: "paxos": { 2024-08-18T21:43:07.470 INFO:tasks.workunit.client.0.smithi099.stdout: "first_committed": 1, 2024-08-18T21:43:07.470 INFO:tasks.workunit.client.0.smithi099.stdout: "last_committed": 695, 2024-08-18T21:43:07.470 INFO:tasks.workunit.client.0.smithi099.stdout: "last_pn": 0, 2024-08-18T21:43:07.470 INFO:tasks.workunit.client.0.smithi099.stdout: "accepted_pn": 0 2024-08-18T21:43:07.470 INFO:tasks.workunit.client.0.smithi099.stdout: } 2024-08-18T21:43:07.470 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:43:07.470 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-08-18T21:43:07.471 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-08-18T21:43:07.471 INFO:tasks.workunit.client.0.smithi099.stdout:delay recovery 2024-08-18T21:43:07.471 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:43:07.796 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:43:07.796 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-18T21:42:33.380006+0000 44'102 53:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:05:19.149157+0000 0 0 2024-08-18T21:43:07.796 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:43:07.796 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:43:07.796 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:43:07.811 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:43:07.812 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-08-18T21:43:07.812 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-08-18T21:43:07.812 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-08-18T21:43:07.813 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:43:07.813 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:07.813 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:43:07.813 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.0.asok 2024-08-18T21:43:07.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-08-18T21:43:07.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.0.asok set_recovery_delay 100000 2024-08-18T21:43:07.920 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:43:07.932 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-08-18T21:43:07.933 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-08-18T21:43:07.933 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-08-18T21:43:07.933 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-08-18T21:43:07.935 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:43:07.935 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:07.936 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:43:07.936 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.2.asok 2024-08-18T21:43:07.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-08-18T21:43:07.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.2.asok set_recovery_delay 100000 2024-08-18T21:43:08.033 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:43:08.044 INFO:tasks.workunit.client.0.smithi099.stdout:revive divergent 1 2024-08-18T21:43:08.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-08-18T21:43:08.045 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:43:08.444 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:43:08.444 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-18T21:42:33.380006+0000 44'102 53:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:05:19.149157+0000 0 0 2024-08-18T21:43:08.444 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:43:08.444 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:43:08.444 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:43:08.459 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-08-18T21:43:09.124 INFO:tasks.workunit.client.0.smithi099.stderr:noup is set 2024-08-18T21:43:09.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-08-18T21:43:09.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:43:09.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:43:09.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:43:09.149 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:43:09.149 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:43:09.149 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:43:09.149 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:43:09.149 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:09.149 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:43:09.150 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:43:09.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:43:09.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:43:09.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:43:09.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:43:09.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:43:09.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:43:09.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:43:09.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:43:09.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:43:09.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:43:09.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-18T21:43:09.151 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:43:09.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-18T21:43:09.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:43:09.152 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-18T21:43:09.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-18T21:43:09.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:43:09.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:43:09.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:43:09.191 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:09.195+0000 7fb5d28cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:09.194 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:09.195+0000 7fb5d28cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:09.195 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:09.199+0000 7fb5d28cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:09.640 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-08-18T21:43:09.640 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-08-18T21:43:10.234 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:10.235+0000 7fb5d28cf7c0 -1 Falling back to public interface 2024-08-18T21:43:12.916 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:12.919+0000 7fb5d28cf7c0 -1 osd.1 30 log_to_monitors true 2024-08-18T21:43:14.641 INFO:tasks.workunit.client.0.smithi099.stdout:delay recovery divergent 2024-08-18T21:43:14.642 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-08-18T21:43:14.642 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:43:15.068 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:43:15.069 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-18T21:42:33.380006+0000 44'102 53:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:05:19.149157+0000 0 0 2024-08-18T21:43:15.069 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:43:15.069 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:43:15.069 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:43:15.085 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-08-18T21:43:15.085 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-08-18T21:43:15.085 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-08-18T21:43:15.086 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:43:15.086 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:15.086 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:43:15.086 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.27423/ceph-osd.1.asok 2024-08-18T21:43:15.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-08-18T21:43:15.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.27423/ceph-osd.1.asok set_recovery_delay 100000 2024-08-18T21:43:15.185 INFO:tasks.workunit.client.0.smithi099.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-08-18T21:43:15.196 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-08-18T21:43:16.052 INFO:tasks.workunit.client.0.smithi099.stderr:noup is unset 2024-08-18T21:43:16.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-08-18T21:43:16.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:43:16.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:43:16.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:43:16.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:43:16.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:43:16.075 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:43:16.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:43:16.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:43:16.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:43:16.608 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 47 up_thru 51 down_at 46 last_clean_interval [37,44) [v2:127.0.0.1:6802/1388599734,v1:127.0.0.1:6803/1388599734] [v2:127.0.0.1:6804/1388599734,v1:127.0.0.1:6805/1388599734] exists,up b0d22869-f4e5-4905-ba11-85fef4467cd0 2024-08-18T21:43:16.610 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:43:16.610 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:43:16.610 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:43:16.610 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-08-18T21:43:16.610 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:43:16.610 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:43:16.611 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:43:16.611 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:43:16.611 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:43:16.611 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:43:16.611 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:43:16.611 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:43:16.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:43:17.083 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/2954765346,v1:127.0.0.1:6819/2954765346] [v2:127.0.0.1:6820/2954765346,v1:127.0.0.1:6821/2954765346] exists,up cf5d0225-bc36-4ab3-94ae-5d9e0c83d536 2024-08-18T21:43:17.085 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:43:17.085 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:43:17.085 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:43:17.085 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-08-18T21:43:17.085 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:43:17.085 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:43:17.086 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:43:17.086 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:43:17.086 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:43:17.086 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:43:17.086 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:43:17.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:43:17.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:43:17.514 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 51 up_thru 0 down_at 50 last_clean_interval [39,44) [v2:127.0.0.1:6810/1576157991,v1:127.0.0.1:6811/1576157991] [v2:127.0.0.1:6812/1576157991,v1:127.0.0.1:6813/1576157991] exists,up a517c788-2442-48b9-984a-35f904f61dbf 2024-08-18T21:43:17.515 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:43:17.515 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:43:17.515 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:43:17.515 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:43:17.920 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:43:17.920 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-18T21:42:33.380006+0000 44'102 56:150 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:05:19.149157+0000 0 0 2024-08-18T21:43:17.920 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:43:17.920 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:43:17.921 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:43:17.935 INFO:tasks.workunit.client.0.smithi099.stdout:wait for peering 2024-08-18T21:43:17.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-08-18T21:43:17.936 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:43:18.332 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:43:18.333 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-18T21:42:33.380006+0000 44'102 56:150 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:05:19.149157+0000 0 0 2024-08-18T21:43:18.333 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:43:18.333 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:43:18.333 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:43:18.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-08-18T21:43:18.463 INFO:tasks.workunit.client.0.smithi099.stdout:killing divergent 1 2024-08-18T21:43:18.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-08-18T21:43:18.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:43:18.875 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:43:18.875 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-18T21:42:33.380006+0000 44'102 56:150 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:05:19.149157+0000 0 0 2024-08-18T21:43:18.875 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:43:18.875 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:43:18.875 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:43:18.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-08-18T21:43:18.891 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:43:18.891 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:43:18.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:43:18.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:43:18.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:43:18.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:43:18.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-08-18T21:43:18.998 INFO:tasks.workunit.client.0.smithi099.stdout:reviving divergent 1 2024-08-18T21:43:18.999 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:43:19.398 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:43:19.399 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-08-18T21:42:33.380006+0000 44'102 56:150 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-20T06:05:19.149157+0000 0 0 2024-08-18T21:43:19.399 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:43:19.399 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:43:19.399 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:43:19.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-08-18T21:43:19.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-08-18T21:43:19.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:43:19.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-18T21:43:19.415 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:43:19.416 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:43:19.418 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:43:19.418 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:19.418 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:43:19.418 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:43:19.418 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' 2024-08-18T21:43:19.418 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:43:19.418 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-08-18T21:43:19.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-08-18T21:43:19.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:43:19.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:43:19.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:43:19.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:43:19.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:43:19.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:43:19.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-08-18T21:43:19.420 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-18T21:43:19.420 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:43:19.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=d6dec24a-9728-40c3-9343-6a2e4528b186 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.27423/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:43:19.421 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-08-18T21:43:19.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-18T21:43:19.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:43:19.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:43:19.422 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:43:19.455 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:19.459+0000 7f8b63f617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:19.457 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:19.459+0000 7f8b63f617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:19.458 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:19.459+0000 7f8b63f617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:19.842 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-18T21:43:19.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:43:19.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:43:19.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:43:19.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:43:19.843 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:43:19.843 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:43:19.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:43:19.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:43:19.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:43:20.289 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 58 up_thru 58 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/2954765346,v1:127.0.0.1:6819/2954765346] [v2:127.0.0.1:6820/2954765346,v1:127.0.0.1:6821/2954765346] exists,up cf5d0225-bc36-4ab3-94ae-5d9e0c83d536 2024-08-18T21:43:20.290 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:43:20.290 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:43:20.290 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:43:20.290 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-08-18T21:43:20.485 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:20.487+0000 7f8b63f617c0 -1 Falling back to public interface 2024-08-18T21:43:23.556 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:23.559+0000 7f8b63f617c0 -1 osd.1 59 log_to_monitors true 2024-08-18T21:43:40.292 INFO:tasks.workunit.client.0.smithi099.stdout:allowing recovery 2024-08-18T21:43:40.292 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-08-18T21:43:40.292 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:43:40.711 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:43:40.711 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-08-18T21:43:25.651467+0000 59'103 64:161 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-19T23:15:59.443030+0000 0 0 2024-08-18T21:43:40.711 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:43:40.711 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:43:40.711 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:43:40.726 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-08-18T21:43:40.726 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-08-18T21:43:40.880 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:43:40.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-08-18T21:43:40.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-08-18T21:43:41.035 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:43:41.048 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-08-18T21:43:41.048 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-08-18T21:43:41.195 INFO:tasks.workunit.client.0.smithi099.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-08-18T21:43:41.209 INFO:tasks.workunit.client.0.smithi099.stdout:reading divergent objects 2024-08-18T21:43:41.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-08-18T21:43:41.209 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-08-18T21:43:41.618 INFO:tasks.workunit.client.0.smithi099.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-08-18T21:43:41.618 INFO:tasks.workunit.client.0.smithi099.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-08-18T21:43:25.651467+0000 59'103 64:161 [1,0,2] 1 [1,0,2] 1 0'0 2024-08-18T21:40:21.170632+0000 0'0 2024-08-18T21:40:21.170632+0000 0 0 periodic scrub scheduled @ 2024-08-19T23:15:59.443030+0000 0 0 2024-08-18T21:43:41.619 INFO:tasks.workunit.client.0.smithi099.stdout: 2024-08-18T21:43:41.619 INFO:tasks.workunit.client.0.smithi099.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-08-18T21:43:41.619 INFO:tasks.workunit.client.0.smithi099.stderr:dumped pgs 2024-08-18T21:43:41.635 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-08-18T21:43:41.636 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-08-18T21:43:41.636 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:43:41.637 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-08-18T21:43:41.671 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:43:41.671 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-08-18T21:43:41.705 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:43:41.706 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-08-18T21:43:41.741 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-08-18T21:43:41.741 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-08-18T21:43:41.776 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-08-18T21:43:41.777 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-08-18T21:43:41.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-18T21:43:41.783 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-08-18T21:41:03.206051+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:43:41.783 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-08-18T21:41:03.206051+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-08-18T21:43:41.783 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:43:41.783 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2024-08-18T21:43:41.783 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-08-18T21:43:41.783 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-08-18T21:43:41.783 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4661.0:1 2024-08-18T21:41:11.580654+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-18T21:43:41.783 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4661.0:1 2024-08-18T21:41:11.580654+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-18T21:43:41.783 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:43:41.783 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2024-08-18T21:43:41.784 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-08-18T21:43:41.784 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-08-18T21:43:41.784 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2024-08-18T21:43:41.784 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4661.0:1 2024-08-18T21:41:11.580654+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-08-18T21:43:41.784 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-08-18T21:43:41.784 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-08-18T21:41:10.322368+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4657.0:1 2024-08-18T21:41:11.330193+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:43:41.785 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-08-18T21:41:10.322368+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:43:41.785 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4657.0:1 2024-08-18T21:41:11.330193+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:43:41.785 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-08-18T21:43:41.785 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2024-08-18T21:43:41.785 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-08-18T21:43:41.785 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-08-18T21:43:41.785 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2024-08-18T21:43:41.785 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4657.0:1 2024-08-18T21:41:11.330193+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-08-18T21:43:41.785 INFO:tasks.workunit.client.0.smithi099.stdout:td/divergent-priors/osd.1.log:2024-08-18T21:43:16.999+0000 7fb5b6859640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-08-18T21:43:41.786 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-08-18T21:43:41.786 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-08-18T21:43:41.792 INFO:tasks.workunit.client.0.smithi099.stdout:success 2024-08-18T21:43:41.792 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-08-18T21:43:41.792 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-08-18T21:43:41.792 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-08-18T21:43:41.792 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-08-18T21:43:42.388 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'test' does not exist 2024-08-18T21:43:42.403 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-08-18T21:43:42.404 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:43:42.404 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:43:42.404 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:43:42.404 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:43:42.404 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:43:47.722 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:43:47.722 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-08-18T21:43:47.722 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-18T21:43:47.722 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:43:47.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-18T21:43:47.723 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:43:47.723 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:43:47.723 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:43:47.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:43:47.724 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:43:47.732 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:43:47.732 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:43:47.733 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:43:47.733 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:43:47.734 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:43:47.734 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:43:47.735 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:43:47.736 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:43:47.736 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:43:47.736 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:43:47.736 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:43:47.737 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:43:47.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:43:47.739 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-18T21:43:47.766 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:43:47.766 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:47.766 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:43:47.766 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27423 2024-08-18T21:43:47.767 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:43:47.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:43:47.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: main: code=0 2024-08-18T21:43:47.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: teardown td/divergent-priors 0 2024-08-18T21:43:47.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-08-18T21:43:47.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-18T21:43:47.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-08-18T21:43:47.768 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:43:47.768 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:43:47.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:43:47.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:43:47.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:43:47.770 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:43:47.770 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:43:47.771 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:43:47.772 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:43:47.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:43:47.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:43:47.773 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:43:47.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:43:47.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:43:47.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:43:47.775 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:43:47.775 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:43:47.776 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-18T21:43:47.776 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-08-18T21:43:47.778 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:43:47.778 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:47.778 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.27423 2024-08-18T21:43:47.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.27423 2024-08-18T21:43:47.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:43:47.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:43:47.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: main: return 0 2024-08-18T21:43:47.780 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-18T21:43:47.780 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-18T21:43:47.833 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-08-18T21:43:47.833 DEBUG:teuthology.orchestra.run.smithi099:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-08-18T21:43:47.888 INFO:tasks.workunit.client.0.smithi099.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-18T21:43:47.893 INFO:tasks.workunit.client.0.smithi099.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-18T21:43:47.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-18T21:43:47.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-18T21:43:47.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export PYTHONWARNINGS=ignore 2024-08-18T21:43:47.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: PYTHONWARNINGS=ignore 2024-08-18T21:43:47.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: export CEPH_CONF=/dev/null 2024-08-18T21:43:47.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: CEPH_CONF=/dev/null 2024-08-18T21:43:47.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: unset CEPH_ARGS 2024-08-18T21:43:47.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: local code 2024-08-18T21:43:47.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: run td/ec-error-rollforward 2024-08-18T21:43:47.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-08-18T21:43:47.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-08-18T21:43:47.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-08-18T21:43:47.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-08-18T21:43:47.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-08-18T21:43:47.894 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-08-18T21:43:47.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none ' 2024-08-18T21:43:47.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-08-18T21:43:47.896 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-08-18T21:43:47.896 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-08-18T21:43:47.896 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-08-18T21:43:47.896 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-08-18T21:43:47.896 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-08-18T21:43:47.896 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-08-18T21:43:47.896 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-08-18T21:43:47.896 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-18T21:43:47.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-08-18T21:43:47.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-08-18T21:43:47.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-08-18T21:43:47.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2024-08-18T21:43:47.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2024-08-18T21:43:47.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-08-18T21:43:47.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:43:47.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-08-18T21:43:47.899 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:43:47.899 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:43:47.899 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:43:47.899 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:43:47.899 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:43:47.900 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:43:47.901 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:43:47.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:43:47.902 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:43:47.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:43:47.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:43:47.903 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:43:47.904 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:43:47.904 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:43:47.905 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:43:47.905 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:43:47.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:43:47.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:43:47.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-08-18T21:43:47.908 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:43:47.908 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:47.909 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:43:47.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85346 2024-08-18T21:43:47.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:43:47.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:43:47.910 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2024-08-18T21:43:47.911 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-18T21:43:47.911 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:47.911 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:43:47.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.85346 2024-08-18T21:43:47.913 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-18T21:43:47.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-18T21:43:47.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-18T21:43:47.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-18T21:43:47.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-08-18T21:43:47.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-08-18T21:43:47.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-08-18T21:43:47.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-08-18T21:43:47.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2024-08-18T21:43:47.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-18T21:43:47.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-18T21:43:47.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-18T21:43:47.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2024-08-18T21:43:47.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-08-18T21:43:48.382 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-18T21:43:48.383 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:43:48.383 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:43:48.383 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:43:48.383 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:48.383 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:43:48.383 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:43:48.383 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-18T21:43:48.599 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-18T21:43:48.601 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-18T21:43:48.601 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85346/ceph-mon.a.asok 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:43:48.602 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85346/ceph-mon.a.asok config get fsid 2024-08-18T21:43:48.717 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-18T21:43:48.718 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:43:48.718 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:43:48.718 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-18T21:43:48.719 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:43:48.719 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:43:48.719 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:43:48.719 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-18T21:43:48.719 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:43:48.719 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:48.719 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:43:48.719 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.85346/ceph-mon.a.asok 2024-08-18T21:43:48.720 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:43:48.720 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.85346/ceph-mon.a.asok config get mon_host 2024-08-18T21:43:48.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-08-18T21:43:48.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2024-08-18T21:43:48.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-18T21:43:48.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-18T21:43:48.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-18T21:43:48.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2024-08-18T21:43:48.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-18T21:43:49.739 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-18T21:43:49.739 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:43:49.740 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:43:49.740 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:43:49.740 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:49.740 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:43:49.740 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:43:49.741 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:43:49.742 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:43:49.777 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:43:49.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-18T21:43:49.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-18T21:43:49.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:43:49.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:43:49.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:43:49.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:43:49.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:43:49.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:43:49.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2024-08-18T21:43:49.781 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:43:49.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=310db5bb-47b0-47ca-98f6-916cfd5b2292 2024-08-18T21:43:49.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 310db5bb-47b0-47ca-98f6-916cfd5b2292' 2024-08-18T21:43:49.781 INFO:tasks.workunit.client.0.smithi099.stdout:add osd0 310db5bb-47b0-47ca-98f6-916cfd5b2292 2024-08-18T21:43:49.781 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:43:49.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVa8JmYmHALxAA1fPXIwe3lr8Sh7HbjatUcQ== 2024-08-18T21:43:49.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVa8JmYmHALxAA1fPXIwe3lr8Sh7HbjatUcQ=="}' 2024-08-18T21:43:49.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 310db5bb-47b0-47ca-98f6-916cfd5b2292 -i td/ec-error-rollforward/0/new.json 2024-08-18T21:43:50.088 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:43:50.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2024-08-18T21:43:50.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVa8JmYmHALxAA1fPXIwe3lr8Sh7HbjatUcQ== --osd-uuid 310db5bb-47b0-47ca-98f6-916cfd5b2292 2024-08-18T21:43:50.130 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:50.131+0000 7fcc6206a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:50.157 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:50.159+0000 7fcc6206a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:50.158 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:50.159+0000 7fcc6206a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:50.158 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:50.163+0000 7fcc6206a7c0 -1 bdev(0x55b65fb8dc00 td/ec-error-rollforward/0/block) open stat got: (1) Operation not permitted 2024-08-18T21:43:50.158 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:50.163+0000 7fcc6206a7c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-08-18T21:43:56.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-08-18T21:43:56.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:43:56.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-18T21:43:56.663 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd0 key to auth repository 2024-08-18T21:43:56.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:43:57.233 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:43:57.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-18T21:43:57.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:43:57.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:43:57.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:43:57.234 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:43:57.263 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:57.267+0000 7fa0a3f5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:57.265 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:57.267+0000 7fa0a3f5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:57.267 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:57.271+0000 7fa0a3f5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:43:57.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-18T21:43:57.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:43:57.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:43:57.661 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:43:57.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:43:57.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:43:57.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:43:57.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:43:57.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:43:57.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:43:58.115 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:43:58.545 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:43:58.547+0000 7fa0a3f5f7c0 -1 Falling back to public interface 2024-08-18T21:43:59.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:43:59.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:43:59.117 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:43:59.118 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:43:59.118 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:43:59.118 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:43:59.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:00.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:00.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:00.535 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:44:00.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:44:00.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:00.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:00.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:01.444 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:44:01.446+0000 7fa0a3f5f7c0 -1 osd.0 0 log_to_monitors true 2024-08-18T21:44:01.975 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:01.975 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:01.976 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:44:01.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:44:01.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:01.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:02.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:03.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:03.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:03.619 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:44:03.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:44:03.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:03.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:04.087 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:05.088 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:05.089 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:05.089 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:44:05.089 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:44:05.089 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:05.090 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:05.609 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:06.611 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:06.611 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:44:06.611 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:06.611 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:44:06.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:06.612 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:07.061 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:08.063 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:08.063 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:08.063 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:44:08.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:44:08.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:08.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:08.517 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:09.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:09.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:09.519 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:44:09.520 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:44:09.520 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:09.520 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:09.994 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:10.995 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:10.995 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:10.996 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:44:10.996 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:44:10.996 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:10.996 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:11.487 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:12.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:12.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:12.489 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:44:12.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:44:12.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:12.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:12.958 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:13.959 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:13.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:13.960 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:44:13.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:44:13.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:13.961 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:14.434 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:15.435 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:15.436 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:15.436 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:44:15.436 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:44:15.437 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:15.437 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:16.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:17.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:17.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:17.099 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:44:17.100 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:44:17.100 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:17.100 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:17.961 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:18.963 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:18.963 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:18.963 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:44:18.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:44:18.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:18.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:19.856 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:20.857 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:20.858 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:20.858 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:44:20.858 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:44:20.858 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:20.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:21.765 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:22.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:22.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:22.768 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:44:22.769 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:44:22.769 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:22.769 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:23.792 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:24.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:24.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:24.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:44:24.795 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:44:24.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:24.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:25.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:26.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:26.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:26.804 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:44:26.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:44:26.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:26.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:44:27.249 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4192316245,v1:127.0.0.1:6803/4192316245] [v2:127.0.0.1:6804/4192316245,v1:127.0.0.1:6805/4192316245] exists,up 310db5bb-47b0-47ca-98f6-916cfd5b2292 2024-08-18T21:44:27.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:44:27.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:44:27.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:44:27.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-08-18T21:44:27.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-08-18T21:44:27.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:44:27.253 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:44:27.254 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:44:27.255 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:44:27.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:44:27.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:44:27.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:44:27.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:44:27.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2024-08-18T21:44:27.256 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:44:27.256 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7647af1e-0c3a-4086-b80d-9177433f89b7 2024-08-18T21:44:27.257 INFO:tasks.workunit.client.0.smithi099.stdout:add osd1 7647af1e-0c3a-4086-b80d-9177433f89b7 2024-08-18T21:44:27.257 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7647af1e-0c3a-4086-b80d-9177433f89b7' 2024-08-18T21:44:27.257 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:44:27.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA7a8Jm3QOSEBAAB6KDzQjLRxVB6/KuyQs+zQ== 2024-08-18T21:44:27.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA7a8Jm3QOSEBAAB6KDzQjLRxVB6/KuyQs+zQ=="}' 2024-08-18T21:44:27.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7647af1e-0c3a-4086-b80d-9177433f89b7 -i td/ec-error-rollforward/1/new.json 2024-08-18T21:44:27.756 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:44:27.771 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2024-08-18T21:44:27.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA7a8Jm3QOSEBAAB6KDzQjLRxVB6/KuyQs+zQ== --osd-uuid 7647af1e-0c3a-4086-b80d-9177433f89b7 2024-08-18T21:44:27.807 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:44:27.810+0000 7fb1cfff37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:44:27.809 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:44:27.810+0000 7fb1cfff37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:44:27.810 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:44:27.814+0000 7fb1cfff37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:44:27.811 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:44:27.814+0000 7fb1cfff37c0 -1 bdev(0x55d838ba0800 td/ec-error-rollforward/1/block) open stat got: (1) Operation not permitted 2024-08-18T21:44:27.811 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:44:27.814+0000 7fb1cfff37c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-08-18T21:44:34.273 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-08-18T21:44:34.273 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:44:34.275 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd1 key to auth repository 2024-08-18T21:44:34.276 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-18T21:44:34.276 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:44:34.843 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:44:34.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-18T21:44:34.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:44:34.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:44:34.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:44:34.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:44:34.873 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:44:34.874+0000 7fa08f13a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:44:34.891 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:44:34.894+0000 7fa08f13a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:44:34.893 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:44:34.894+0000 7fa08f13a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:44:35.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-18T21:44:35.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:44:35.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:44:35.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:44:35.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:44:35.271 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:35.272 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:44:35.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:44:35.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:35.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:35.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:35.997 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:44:35.998+0000 7fa08f13a7c0 -1 Falling back to public interface 2024-08-18T21:44:36.680 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:36.680 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:36.680 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:44:36.681 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:44:36.681 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:36.681 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:37.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:38.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:38.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:38.150 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:44:38.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:44:38.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:38.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:38.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:38.958 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:44:38.962+0000 7fa08f13a7c0 -1 osd.1 0 log_to_monitors true 2024-08-18T21:44:39.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:39.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:39.623 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:44:39.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:44:39.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:39.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:40.073 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:41.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:41.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:41.075 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:44:41.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:44:41.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:41.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:41.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:42.602 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:42.602 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:42.602 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:44:42.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:44:42.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:42.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:43.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:44.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:44.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:44.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:44:44.066 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:44:44.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:44.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:44.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:45.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:45.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:45.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:44:45.504 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:44:45.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:45.505 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:46.039 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:47.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:47.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:47.040 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:44:47.041 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:44:47.041 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:47.041 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:47.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:48.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:48.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:48.483 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:44:48.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:44:48.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:48.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:49.052 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:50.054 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:50.054 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:50.055 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:44:50.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:44:50.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:50.055 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:50.514 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:51.516 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:51.516 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:51.516 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:44:51.517 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:44:51.517 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:51.517 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:52.023 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:53.025 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:53.025 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:53.025 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:44:53.026 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:44:53.026 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:53.026 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:53.818 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:54.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:54.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:54.820 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:44:54.821 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:44:54.821 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:54.821 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:55.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:56.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:56.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:56.976 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:44:56.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:44:56.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:56.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:57.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:44:58.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:44:58.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:44:58.892 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:44:58.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:44:58.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:44:58.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:44:59.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:00.764 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:00.764 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:00.764 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:45:00.765 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:45:00.765 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:00.765 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:45:01.598 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:02.599 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:02.599 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:02.599 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:45:02.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:45:02.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:02.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:45:03.505 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:04.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:04.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:04.507 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:45:04.508 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:45:04.508 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:04.508 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:45:05.092 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2711287165,v1:127.0.0.1:6811/2711287165] [v2:127.0.0.1:6812/2711287165,v1:127.0.0.1:6813/2711287165] exists,up 7647af1e-0c3a-4086-b80d-9177433f89b7 2024-08-18T21:45:05.094 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:45:05.094 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:45:05.094 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:45:05.094 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-08-18T21:45:05.094 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-08-18T21:45:05.094 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:45:05.094 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-18T21:45:05.094 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:45:05.095 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:45:05.096 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:45:05.096 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:45:05.096 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:45:05.097 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:45:05.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:45:05.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:45:05.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:45:05.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:45:05.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-18T21:45:05.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-18T21:45:05.098 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:45:05.098 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:45:05.098 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:45:05.098 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:45:05.098 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:45:05.098 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:45:05.098 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2024-08-18T21:45:05.098 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:45:05.098 INFO:tasks.workunit.client.0.smithi099.stdout:add osd2 3b5a7dd7-439a-435b-8a01-bd5c9de30360 2024-08-18T21:45:05.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3b5a7dd7-439a-435b-8a01-bd5c9de30360 2024-08-18T21:45:05.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3b5a7dd7-439a-435b-8a01-bd5c9de30360' 2024-08-18T21:45:05.099 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:45:05.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBha8JmvUhxBxAAjPrhPOs5li95ViNWMNc3iQ== 2024-08-18T21:45:05.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBha8JmvUhxBxAAjPrhPOs5li95ViNWMNc3iQ=="}' 2024-08-18T21:45:05.119 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3b5a7dd7-439a-435b-8a01-bd5c9de30360 -i td/ec-error-rollforward/2/new.json 2024-08-18T21:45:05.609 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:45:05.625 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2024-08-18T21:45:05.626 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBha8JmvUhxBxAAjPrhPOs5li95ViNWMNc3iQ== --osd-uuid 3b5a7dd7-439a-435b-8a01-bd5c9de30360 2024-08-18T21:45:05.650 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:05.653+0000 7f0df63d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:05.652 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:05.653+0000 7f0df63d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:05.653 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:05.653+0000 7f0df63d57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:05.654 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:05.657+0000 7f0df63d57c0 -1 bdev(0x5593f2fb5c00 td/ec-error-rollforward/2/block) open stat got: (1) Operation not permitted 2024-08-18T21:45:05.654 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:05.657+0000 7f0df63d57c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-08-18T21:45:12.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-08-18T21:45:12.837 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:45:12.838 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-18T21:45:12.839 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd2 key to auth repository 2024-08-18T21:45:12.839 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:45:13.415 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:45:13.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-18T21:45:13.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:45:13.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:45:13.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:45:13.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:45:13.446 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:13.445+0000 7fc0beb407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:13.489 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:13.489+0000 7fc0beb407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:13.491 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:13.493+0000 7fc0beb407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:13.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-18T21:45:13.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:45:13.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:45:13.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:45:13.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:45:13.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:13.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:45:13.890 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:45:13.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:13.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:14.304 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:14.705 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:14.709+0000 7fc0beb407c0 -1 Falling back to public interface 2024-08-18T21:45:15.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:15.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:15.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:45:15.306 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:45:15.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:15.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:15.729 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:16.731 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:16.731 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:16.731 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:45:16.731 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:45:16.732 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:16.732 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:17.178 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:17.674 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:17.677+0000 7fc0beb407c0 -1 osd.2 0 log_to_monitors true 2024-08-18T21:45:18.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:18.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:18.180 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:45:18.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:45:18.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:18.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:18.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:19.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:19.860 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:19.860 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:45:19.860 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:45:19.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:19.861 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:20.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:21.335 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:21.335 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:21.335 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:45:21.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:45:21.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:21.336 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:21.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:22.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:22.869 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:22.869 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:45:22.869 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:45:22.870 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:22.870 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:23.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:24.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:24.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:24.348 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:45:24.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:45:24.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:24.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:24.910 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:25.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:25.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:25.912 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:45:25.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:45:25.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:25.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:26.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:27.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:27.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:27.351 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:45:27.351 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:45:27.351 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:27.351 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:27.820 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:28.821 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:28.821 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:28.822 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:45:28.822 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:45:28.822 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:28.822 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:29.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:30.273 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:30.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:30.274 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:45:30.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:45:30.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:30.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:30.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:31.908 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:31.908 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:31.909 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:45:31.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:45:31.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:31.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:32.391 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:33.392 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:33.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:33.393 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:45:33.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:45:33.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:33.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:34.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:35.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:35.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:35.205 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:45:35.205 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:45:35.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:35.206 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:35.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:36.754 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:36.755 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:36.755 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:45:36.755 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:45:36.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:36.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:37.740 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:38.742 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:38.742 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:38.743 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:45:38.743 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:45:38.743 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:38.743 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:39.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:40.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:40.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:40.835 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:45:40.836 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:45:40.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:40.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:41.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:42.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:42.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:42.622 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:45:42.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:45:42.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:42.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:43.410 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:44.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:44.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:44.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-18T21:45:44.413 INFO:tasks.workunit.client.0.smithi099.stdout:19 2024-08-18T21:45:44.413 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:44.413 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:45:44.875 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2563045498,v1:127.0.0.1:6819/2563045498] [v2:127.0.0.1:6820/2563045498,v1:127.0.0.1:6821/2563045498] exists,up 3b5a7dd7-439a-435b-8a01-bd5c9de30360 2024-08-18T21:45:44.877 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:45:44.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:45:44.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:45:44.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-08-18T21:45:44.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-08-18T21:45:44.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:45:44.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-18T21:45:44.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:45:44.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2024-08-18T21:45:44.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-18T21:45:44.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:45:44.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:45:44.879 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:45:44.880 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2024-08-18T21:45:44.881 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:45:44.881 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd29ba4f-a51c-4d09-9425-eeaa07a9bcc7 2024-08-18T21:45:44.881 INFO:tasks.workunit.client.0.smithi099.stdout:add osd3 cd29ba4f-a51c-4d09-9425-eeaa07a9bcc7 2024-08-18T21:45:44.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 cd29ba4f-a51c-4d09-9425-eeaa07a9bcc7' 2024-08-18T21:45:44.882 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:45:44.897 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIa8Jmz/rSNRAAJzKSlfvH8JLWm6cvu8BChg== 2024-08-18T21:45:44.897 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIa8Jmz/rSNRAAJzKSlfvH8JLWm6cvu8BChg=="}' 2024-08-18T21:45:44.897 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd29ba4f-a51c-4d09-9425-eeaa07a9bcc7 -i td/ec-error-rollforward/3/new.json 2024-08-18T21:45:45.347 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:45:45.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2024-08-18T21:45:45.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIa8Jmz/rSNRAAJzKSlfvH8JLWm6cvu8BChg== --osd-uuid cd29ba4f-a51c-4d09-9425-eeaa07a9bcc7 2024-08-18T21:45:45.396 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:45.400+0000 7f2e931f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:45.412 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:45.416+0000 7f2e931f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:45.413 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:45.416+0000 7f2e931f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:45.413 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:45.416+0000 7f2e931f47c0 -1 bdev(0x556ca2f1fc00 td/ec-error-rollforward/3/block) open stat got: (1) Operation not permitted 2024-08-18T21:45:45.414 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:45.416+0000 7f2e931f47c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-08-18T21:45:52.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-08-18T21:45:52.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:45:52.240 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd3 key to auth repository 2024-08-18T21:45:52.241 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-18T21:45:52.241 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:45:52.815 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-18T21:45:52.816 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.3 2024-08-18T21:45:52.816 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:45:52.816 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:45:52.816 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:45:52.817 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:45:52.846 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:52.848+0000 7fa6be9417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:52.858 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:52.864+0000 7fa6be9417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:52.860 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:52.864+0000 7fa6be9417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:45:53.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-18T21:45:53.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:45:53.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-18T21:45:53.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:45:53.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:45:53.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:53.229 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:45:53.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:45:53.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:53.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:45:53.705 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:54.125 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:54.128+0000 7fa6be9417c0 -1 Falling back to public interface 2024-08-18T21:45:54.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:54.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:54.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:45:54.708 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:45:54.708 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:54.708 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:45:55.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:56.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:56.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:56.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:45:56.135 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:45:56.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:56.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:45:56.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:57.099 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:45:57.088+0000 7fa6be9417c0 -1 osd.3 0 log_to_monitors true 2024-08-18T21:45:57.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:57.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:57.625 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:45:57.625 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:45:57.625 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:57.625 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:45:58.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:45:59.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:45:59.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:45:59.152 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:45:59.152 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:45:59.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:45:59.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:45:59.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:00.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:00.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:00.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:46:00.623 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:46:00.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:00.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:01.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:02.214 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:02.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:02.215 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:46:02.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:46:02.216 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:02.216 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:02.708 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:03.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:03.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:03.710 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:46:03.710 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:46:03.710 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:03.710 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:04.224 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:05.225 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:05.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:05.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:46:05.226 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:46:05.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:05.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:05.693 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:06.695 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:06.695 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:06.695 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:46:06.696 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:46:06.696 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:06.696 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:07.213 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:08.214 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:08.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:08.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:46:08.215 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:46:08.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:08.215 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:08.680 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:09.682 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:09.683 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:09.683 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:46:09.683 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:46:09.683 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:09.684 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:10.249 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:11.250 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:11.250 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:11.251 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:46:11.252 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:46:11.252 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:11.252 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:11.850 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:12.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:12.852 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:12.852 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:46:12.852 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:46:12.852 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:12.852 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:13.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:14.875 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:14.875 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:14.875 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:46:14.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:46:14.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:14.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:15.702 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:16.704 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:16.704 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:46:16.705 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:16.705 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:46:16.705 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:16.705 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:17.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:18.623 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:18.624 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:46:18.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:18.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:46:18.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:18.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:19.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:20.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:20.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:20.070 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:46:20.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:46:20.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:20.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:20.922 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:21.924 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:21.924 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:21.924 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:46:21.924 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:46:21.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:21.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:22.754 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:23.755 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:23.755 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:23.755 INFO:tasks.workunit.client.0.smithi099.stdout:19 2024-08-18T21:46:23.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-18T21:46:23.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:23.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:24.191 INFO:tasks.workunit.client.0.smithi099.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1619571929,v1:127.0.0.1:6827/1619571929] [v2:127.0.0.1:6828/1619571929,v1:127.0.0.1:6829/1619571929] exists,up cd29ba4f-a51c-4d09-9425-eeaa07a9bcc7 2024-08-18T21:46:24.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:46:24.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:46:24.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:46:24.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-08-18T21:46:24.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-08-18T21:46:25.628 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'ec' already exists 2024-08-18T21:46:25.645 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-08-18T21:46:27.067 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-08-18T21:46:27.068 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 89985 2024-08-18T21:46:27.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=92814 2024-08-18T21:46:27.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-08-18T21:46:27.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-08-18T21:46:28.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 92835' 2024-08-18T21:46:28.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-08-18T21:46:28.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 92836' 2024-08-18T21:46:28.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-08-18T21:46:28.072 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 92837' 2024-08-18T21:46:28.072 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-08-18T21:46:28.072 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-08-18T21:46:29.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-08-18T21:46:29.075 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:46:29.075 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:46:29.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:46:29.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:46:29.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:46:29.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:46:29.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 92814 92835 92836 92837 2024-08-18T21:46:29.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-08-18T21:46:29.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-08-18T21:46:29.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-08-18T21:46:29.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-18T21:46:29.192 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:46:29.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:46:29.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:46:29.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:46:29.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-08-18T21:46:29.194 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:46:29.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-18T21:46:29.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:46:29.195 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-08-18T21:46:29.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-18T21:46:29.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:46:29.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:46:29.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:46:29.219 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:29.223+0000 7fd62696b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:29.222 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:29.223+0000 7fd62696b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:29.226 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:29.227+0000 7fd62696b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:29.654 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-18T21:46:29.654 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:46:29.655 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:46:29.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:46:29.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:46:29.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:46:29.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:29.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:46:29.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:46:29.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:30.130 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4192316245,v1:127.0.0.1:6803/4192316245] [v2:127.0.0.1:6804/4192316245,v1:127.0.0.1:6805/4192316245] exists,up 310db5bb-47b0-47ca-98f6-916cfd5b2292 2024-08-18T21:46:30.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:46:30.132 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-08-18T21:46:30.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-18T21:46:30.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-18T21:46:30.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:46:30.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:46:30.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:46:30.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:46:30.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:46:30.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:46:30.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-08-18T21:46:30.135 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:46:30.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-18T21:46:30.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:46:30.136 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-08-18T21:46:30.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-18T21:46:30.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:46:30.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:46:30.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:46:30.158 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:30.159+0000 7f4d114377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:30.160 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:30.163+0000 7f4d114377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:30.161 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:30.163+0000 7f4d114377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:30.493 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:30.495+0000 7fd62696b7c0 -1 Falling back to public interface 2024-08-18T21:46:30.561 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-18T21:46:30.561 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:46:30.561 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:46:30.561 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:46:30.561 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:46:30.561 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:30.561 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:46:30.562 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:46:30.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:46:30.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:30.999 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2711287165,v1:127.0.0.1:6811/2711287165] [v2:127.0.0.1:6812/2711287165,v1:127.0.0.1:6813/2711287165] exists,up 7647af1e-0c3a-4086-b80d-9177433f89b7 2024-08-18T21:46:31.002 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:46:31.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:46:31.004 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:46:31.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-18T21:46:31.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-18T21:46:31.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:46:31.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:46:31.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:46:31.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:46:31.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:46:31.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:46:31.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-08-18T21:46:31.006 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:46:31.006 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-18T21:46:31.006 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:46:31.007 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-08-18T21:46:31.007 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-18T21:46:31.007 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:46:31.007 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:46:31.007 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:46:31.028 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:31.031+0000 7f6a6b1f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:31.039 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:31.043+0000 7f6a6b1f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:31.040 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:31.043+0000 7f6a6b1f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:31.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-18T21:46:31.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:46:31.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:46:31.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:46:31.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:46:31.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:31.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:46:31.482 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:46:31.482 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:46:31.482 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:31.717 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:31.719+0000 7f4d114377c0 -1 Falling back to public interface 2024-08-18T21:46:31.911 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2563045498,v1:127.0.0.1:6819/2563045498] [v2:127.0.0.1:6820/2563045498,v1:127.0.0.1:6821/2563045498] exists,up 3b5a7dd7-439a-435b-8a01-bd5c9de30360 2024-08-18T21:46:31.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:46:31.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:46:31.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:46:31.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-08-18T21:46:31.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-08-18T21:46:31.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:46:31.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-18T21:46:31.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:46:31.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:46:31.914 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:46:31.915 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:31.915 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:46:31.915 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:46:31.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' 2024-08-18T21:46:31.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:46:31.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-08-18T21:46:31.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-08-18T21:46:31.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:46:31.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:46:31.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:46:31.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:46:31.916 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:46:31.917 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:46:31.917 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-08-18T21:46:31.917 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-18T21:46:31.917 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.3 2024-08-18T21:46:31.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=8b9ca409-758a-4530-b9b5-4071e67cd004 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.85346/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:46:31.918 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-08-18T21:46:31.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-18T21:46:31.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:46:31.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:46:31.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:46:31.949 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:31.951+0000 7f10dadba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:31.959 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:31.963+0000 7f10dadba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:31.961 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:31.963+0000 7f10dadba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:32.297 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:32.299+0000 7f6a6b1f77c0 -1 Falling back to public interface 2024-08-18T21:46:32.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-18T21:46:32.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:46:32.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-18T21:46:32.370 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:46:32.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:46:32.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:46:32.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:32.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:46:32.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:32.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:46:32.881 INFO:tasks.workunit.client.0.smithi099.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1619571929,v1:127.0.0.1:6827/1619571929] [v2:127.0.0.1:6828/1619571929,v1:127.0.0.1:6829/1619571929] exists,up cd29ba4f-a51c-4d09-9425-eeaa07a9bcc7 2024-08-18T21:46:32.881 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:46:32.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:46:32.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:46:32.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-08-18T21:46:32.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:46:32.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:46:32.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:46:32.882 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-18T21:46:32.883 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:46:32.883 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:46:32.883 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:46:32.883 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:46:32.883 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:46:32.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-18T21:46:32.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:46:32.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:46:32.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:46:32.983 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:46:32.983 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:46:33.036 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:33.039+0000 7f10dadba7c0 -1 Falling back to public interface 2024-08-18T21:46:33.389 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:46:33.390 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:46:33.390 INFO:tasks.workunit.client.0.smithi099.stderr:2 2024-08-18T21:46:33.390 INFO:tasks.workunit.client.0.smithi099.stderr:3' 2024-08-18T21:46:33.390 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:46:33.390 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:46:33.390 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:46:34.435 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:34.435+0000 7fd62696b7c0 -1 osd.0 26 log_to_monitors true 2024-08-18T21:46:36.272 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:36.274+0000 7f4d114377c0 -1 osd.1 26 log_to_monitors true 2024-08-18T21:46:36.677 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:36.678+0000 7f6a6b1f77c0 -1 osd.2 25 log_to_monitors true 2024-08-18T21:46:37.265 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:37.266+0000 7f10dadba7c0 -1 osd.3 26 log_to_monitors true 2024-08-18T21:46:38.564 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=124554051587 2024-08-18T21:46:38.564 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 124554051587 2024-08-18T21:46:38.564 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-124554051587' 2024-08-18T21:46:38.564 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:46:38.565 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:46:38.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=141733920771 2024-08-18T21:46:38.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 141733920771 2024-08-18T21:46:38.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-124554051587 1-141733920771' 2024-08-18T21:46:38.725 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:46:38.726 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:46:38.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=141733920771 2024-08-18T21:46:38.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 141733920771 2024-08-18T21:46:38.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-124554051587 1-141733920771 2-141733920771' 2024-08-18T21:46:38.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:46:38.971 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-18T21:46:39.121 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=150323855363 2024-08-18T21:46:39.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 150323855363 2024-08-18T21:46:39.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-124554051587 1-141733920771 2-141733920771 3-150323855363' 2024-08-18T21:46:39.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:46:39.122 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-124554051587 2024-08-18T21:46:39.122 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:46:39.124 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:46:39.124 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-124554051587 2024-08-18T21:46:39.124 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:46:39.125 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=124554051587 2024-08-18T21:46:39.126 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 124554051587 2024-08-18T21:46:39.126 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 124554051587' 2024-08-18T21:46:39.126 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:46:39.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 1 -lt 124554051587 2024-08-18T21:46:39.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:46:40.567 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:46:40.568 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:46:41.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 124554051588 -lt 124554051587 2024-08-18T21:46:41.000 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:46:41.001 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-141733920771 2024-08-18T21:46:41.001 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:46:41.003 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:46:41.003 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-141733920771 2024-08-18T21:46:41.003 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:46:41.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=141733920771 2024-08-18T21:46:41.005 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 141733920771 2024-08-18T21:46:41.006 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 141733920771' 2024-08-18T21:46:41.006 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:46:41.459 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 141733920771 -lt 141733920771 2024-08-18T21:46:41.459 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:46:41.459 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-141733920771 2024-08-18T21:46:41.460 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:46:41.461 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:46:41.462 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-141733920771 2024-08-18T21:46:41.462 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:46:41.463 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=141733920771 2024-08-18T21:46:41.463 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 141733920771 2024-08-18T21:46:41.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2024-08-18T21:46:41.464 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:46:41.922 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 141733920771 -lt 141733920771 2024-08-18T21:46:41.922 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:46:41.923 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:46:41.923 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-150323855363 2024-08-18T21:46:41.924 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=3 2024-08-18T21:46:41.925 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 3-150323855363 2024-08-18T21:46:41.925 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:46:41.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=150323855363 2024-08-18T21:46:41.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2024-08-18T21:46:41.927 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.3 seq 150323855363 2024-08-18T21:46:41.927 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-18T21:46:42.366 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 150323855363 -lt 150323855363 2024-08-18T21:46:42.367 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:46:42.367 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:46:42.367 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:46:42.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-08-18T21:46:42.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:46:42.896 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:46:42.896 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:46:42.896 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:46:42.896 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:46:42.896 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:46:42.896 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:46:43.343 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-08-18T21:46:43.344 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:46:43.344 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:46:43.344 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:46:43.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-08-18T21:46:43.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:46:43.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:46:43.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-08-18T21:46:43.874 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-08-18T21:46:43.874 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:46:43.874 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-08-18T21:46:43.875 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:46:43.875 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:46:43.875 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:46:43.875 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:46:43.875 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:46:44.185 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:46:44.186 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:46:44.186 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:46:44.187 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:46:44.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:46:44.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:46:44.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:46:44.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:46:44.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:46:44.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:46:44.191 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:46:44.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:46:44.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:46:44.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-08-18T21:46:44.215 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:46:44.216 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:44.216 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:46:44.216 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85346 2024-08-18T21:46:44.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:46:44.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:46:44.218 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: main: code=0 2024-08-18T21:46:44.219 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: teardown td/ec-error-rollforward 0 2024-08-18T21:46:44.219 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-08-18T21:46:44.219 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-18T21:46:44.219 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-08-18T21:46:44.219 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:46:44.219 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:46:44.220 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:46:44.220 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:46:44.220 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:46:44.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:46:44.221 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:46:44.221 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:46:44.222 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:46:44.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:46:44.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:46:44.223 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:46:44.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:46:44.224 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:46:44.225 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:46:44.225 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:46:44.225 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:46:44.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-18T21:46:44.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-08-18T21:46:44.227 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:46:44.227 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:44.228 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.85346 2024-08-18T21:46:44.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.85346 2024-08-18T21:46:44.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:46:44.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:46:44.230 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: main: return 0 2024-08-18T21:46:44.231 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-18T21:46:44.231 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-18T21:46:44.283 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-08-18T21:46:44.284 DEBUG:teuthology.orchestra.run.smithi099:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-08-18T21:46:44.340 INFO:tasks.workunit.client.0.smithi099.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-18T21:46:44.346 INFO:tasks.workunit.client.0.smithi099.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-18T21:46:44.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-18T21:46:44.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-18T21:46:44.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export PYTHONWARNINGS=ignore 2024-08-18T21:46:44.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: PYTHONWARNINGS=ignore 2024-08-18T21:46:44.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: export CEPH_CONF=/dev/null 2024-08-18T21:46:44.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: CEPH_CONF=/dev/null 2024-08-18T21:46:44.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: unset CEPH_ARGS 2024-08-18T21:46:44.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: local code 2024-08-18T21:46:44.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: run td/osd-bench 2024-08-18T21:46:44.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-08-18T21:46:44.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-08-18T21:46:44.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-08-18T21:46:44.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-08-18T21:46:44.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-08-18T21:46:44.349 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-08-18T21:46:44.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=9bda1516-ca52-48a6-be7d-3b3d57dc606b --auth-supported=none ' 2024-08-18T21:46:44.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-08-18T21:46:44.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-08-18T21:46:44.357 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-08-18T21:46:44.357 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-18T21:46:44.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-08-18T21:46:44.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-08-18T21:46:44.357 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-08-18T21:46:44.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2024-08-18T21:46:44.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2024-08-18T21:46:44.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-08-18T21:46:44.358 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:46:44.359 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-08-18T21:46:44.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:46:44.359 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:46:44.359 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:46:44.359 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:46:44.360 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:46:44.360 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:46:44.360 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:46:44.360 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:46:44.361 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:46:44.361 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:46:44.361 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:46:44.361 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:46:44.361 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:46:44.361 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:46:44.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:46:44.362 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:46:44.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:46:44.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:46:44.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-08-18T21:46:44.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:46:44.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:44.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95377 2024-08-18T21:46:44.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95377 2024-08-18T21:46:44.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:46:44.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:46:44.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2024-08-18T21:46:44.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-18T21:46:44.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:44.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95377 2024-08-18T21:46:44.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.95377 2024-08-18T21:46:44.372 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-18T21:46:44.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-18T21:46:44.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-18T21:46:44.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-18T21:46:44.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-08-18T21:46:44.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-08-18T21:46:44.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-08-18T21:46:44.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-08-18T21:46:44.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2024-08-18T21:46:44.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-18T21:46:44.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-18T21:46:44.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-18T21:46:44.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2024-08-18T21:46:44.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-08-18T21:46:44.857 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-18T21:46:44.858 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:46:44.858 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:46:44.858 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:46:44.858 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:44.858 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95377 2024-08-18T21:46:44.858 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95377/$cluster-$name.asok' 2024-08-18T21:46:44.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.95377/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-18T21:46:45.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-18T21:46:45.072 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-18T21:46:45.072 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:46:45.072 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:46:45.072 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-18T21:46:45.072 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-18T21:46:45.073 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:46:45.073 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:46:45.074 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:46:45.074 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:46:45.074 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:45.074 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95377 2024-08-18T21:46:45.074 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95377/ceph-mon.a.asok 2024-08-18T21:46:45.074 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:46:45.074 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95377/ceph-mon.a.asok config get fsid 2024-08-18T21:46:45.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-18T21:46:45.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:46:45.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:46:45.190 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-18T21:46:45.191 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-18T21:46:45.191 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:46:45.191 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:46:45.191 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:46:45.191 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:46:45.192 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:45.192 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95377 2024-08-18T21:46:45.192 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95377/ceph-mon.a.asok 2024-08-18T21:46:45.192 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:46:45.192 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95377/ceph-mon.a.asok config get mon_host 2024-08-18T21:46:45.301 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-08-18T21:46:45.301 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2024-08-18T21:46:45.301 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-18T21:46:45.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-18T21:46:45.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-18T21:46:45.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2024-08-18T21:46:45.302 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-18T21:46:46.210 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-18T21:46:46.211 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:46:46.211 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:46:46.211 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:46:46.211 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:46.211 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95377 2024-08-18T21:46:46.211 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95377/$cluster-$name.asok' 2024-08-18T21:46:46.211 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:46:46.212 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.95377/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:46:46.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-08-18T21:46:46.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2024-08-18T21:46:46.247 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9bda1516-ca52-48a6-be7d-3b3d57dc606b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-08-18T21:46:46.248 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95377 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95377/$cluster-$name.asok' 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95377/$cluster-$name.asok' 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-08-18T21:46:46.249 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:46:46.250 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:46:46.250 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:46:46.250 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:46:46.250 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:46:46.250 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:46:46.250 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2024-08-18T21:46:46.250 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:46:46.250 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d560f8b1-9cb1-4bac-ac3d-10288770e34e 2024-08-18T21:46:46.250 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d560f8b1-9cb1-4bac-ac3d-10288770e34e' 2024-08-18T21:46:46.250 INFO:tasks.workunit.client.0.smithi099.stdout:add osd0 d560f8b1-9cb1-4bac-ac3d-10288770e34e 2024-08-18T21:46:46.251 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:46:46.264 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDGa8Jmcm4SEBAASLIsAEqopOlbakU3YyJmsQ== 2024-08-18T21:46:46.264 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDGa8Jmcm4SEBAASLIsAEqopOlbakU3YyJmsQ=="}' 2024-08-18T21:46:46.264 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d560f8b1-9cb1-4bac-ac3d-10288770e34e -i td/osd-bench/0/new.json 2024-08-18T21:46:46.581 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:46:46.601 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2024-08-18T21:46:46.602 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9bda1516-ca52-48a6-be7d-3b3d57dc606b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.95377/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDGa8Jmcm4SEBAASLIsAEqopOlbakU3YyJmsQ== --osd-uuid d560f8b1-9cb1-4bac-ac3d-10288770e34e 2024-08-18T21:46:46.634 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:46.634+0000 7f0657da77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:46.649 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:46.650+0000 7f0657da77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:46.651 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:46.654+0000 7f0657da77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:46.651 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:46.654+0000 7f0657da77c0 -1 bdev(0x556eddcb5c00 td/osd-bench/0/block) open stat got: (1) Operation not permitted 2024-08-18T21:46:46.651 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:46.654+0000 7f0657da77c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-08-18T21:46:53.261 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2024-08-18T21:46:53.262 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:46:53.263 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd0 key to auth repository 2024-08-18T21:46:53.263 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-18T21:46:53.264 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:46:53.838 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:46:53.838 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-18T21:46:53.838 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9bda1516-ca52-48a6-be7d-3b3d57dc606b --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.95377/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:46:53.839 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:46:53.839 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:46:53.839 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:46:53.868 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:53.870+0000 7fbc2fe377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:53.916 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:53.918+0000 7fbc2fe377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:53.917 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:53.918+0000 7fbc2fe377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:46:54.264 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-18T21:46:54.264 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:46:54.264 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:46:54.264 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:46:54.264 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:46:54.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:54.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:46:54.265 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:46:54.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:54.265 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:46:54.726 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:54.966 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:54.966+0000 7fbc2fe377c0 -1 Falling back to public interface 2024-08-18T21:46:55.728 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:55.728 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:55.728 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:46:55.729 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:46:55.729 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:55.729 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:46:56.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:57.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:57.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:57.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:46:57.191 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:46:57.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:57.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:46:57.626 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:46:57.805 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:46:57.806+0000 7fbc2fe377c0 -1 osd.0 0 log_to_monitors true 2024-08-18T21:46:58.628 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:46:58.628 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:46:58.628 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:46:58.629 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:46:58.629 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:46:58.629 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:46:59.153 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:00.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:00.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:00.155 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:47:00.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:47:00.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:00.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:00.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:01.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:01.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:01.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:47:01.621 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:47:01.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:01.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:02.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:03.124 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:03.124 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:03.124 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:47:03.125 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:47:03.125 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:03.125 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:03.563 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:04.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:04.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:04.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:47:04.565 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:47:04.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:04.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:05.145 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:06.146 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:06.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:06.147 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:47:06.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:47:06.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:06.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:06.594 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:07.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:07.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:07.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:47:07.597 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:47:07.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:07.598 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:08.185 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:09.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:09.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:09.188 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:47:09.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:47:09.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:09.188 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:09.631 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:10.633 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:10.633 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:10.633 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:47:10.633 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:47:10.634 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:10.634 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:11.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:12.159 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:12.159 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:47:12.160 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:12.160 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:47:12.160 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:12.160 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:12.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:13.838 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:13.838 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:13.838 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:47:13.838 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:47:13.839 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:13.839 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:14.760 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:15.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:15.762 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:15.762 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:47:15.763 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:47:15.763 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:15.763 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:16.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:17.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:17.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:17.579 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:47:17.580 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:47:17.580 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:17.580 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:18.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:19.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:19.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:19.180 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:47:19.181 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:47:19.181 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:19.181 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:19.970 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:20.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:20.972 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:20.972 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:47:20.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:47:20.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:20.973 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:21.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:22.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:22.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:22.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:47:22.505 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:47:22.505 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:22.505 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:22.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:47:23.978 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:47:23.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:47:23.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-18T21:47:23.979 INFO:tasks.workunit.client.0.smithi099.stdout:19 2024-08-18T21:47:23.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:47:23.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:47:24.402 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/335367119,v1:127.0.0.1:6803/335367119] [v2:127.0.0.1:6804/335367119,v1:127.0.0.1:6805/335367119] exists,up d560f8b1-9cb1-4bac-ac3d-10288770e34e 2024-08-18T21:47:24.403 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:47:24.403 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:47:24.403 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:47:24.404 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-08-18T21:47:24.404 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-08-18T21:47:24.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-08-18T21:47:24.421 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-08-18T21:47:24.422 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-08-18T21:47:24.435 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-08-18T21:47:24.435 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-08-18T21:47:24.435 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-08-18T21:47:24.452 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-08-18T21:47:24.452 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-08-18T21:47:24.452 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-08-18T21:47:24.470 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-08-18T21:47:24.470 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-08-18T21:47:24.470 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: local dir=td/osd-bench 2024-08-18T21:47:24.470 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: shift 2024-08-18T21:47:24.470 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: local expected=osd_bench_max_block_size 2024-08-18T21:47:24.471 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: shift 2024-08-18T21:47:24.471 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: local success 2024-08-18T21:47:24.471 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-08-18T21:47:24.587 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: expect_failure: success=false 2024-08-18T21:47:24.587 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: false 2024-08-18T21:47:24.587 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-08-18T21:47:24.588 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: expect_failure: return 0 2024-08-18T21:47:24.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-08-18T21:47:24.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-08-18T21:47:24.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-08-18T21:47:24.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: local dir=td/osd-bench 2024-08-18T21:47:24.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: shift 2024-08-18T21:47:24.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: local expected=bench_small_size_max_iops 2024-08-18T21:47:24.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: shift 2024-08-18T21:47:24.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: local success 2024-08-18T21:47:24.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-08-18T21:47:24.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: expect_failure: success=false 2024-08-18T21:47:24.708 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: false 2024-08-18T21:47:24.708 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-08-18T21:47:24.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: expect_failure: return 0 2024-08-18T21:47:24.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-08-18T21:47:24.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-08-18T21:47:24.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-08-18T21:47:24.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: local dir=td/osd-bench 2024-08-18T21:47:24.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: shift 2024-08-18T21:47:24.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-08-18T21:47:24.709 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: shift 2024-08-18T21:47:24.710 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: expect_failure: local success 2024-08-18T21:47:24.710 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-08-18T21:47:24.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: expect_failure: success=false 2024-08-18T21:47:24.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: false 2024-08-18T21:47:24.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-08-18T21:47:24.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: expect_failure: return 0 2024-08-18T21:47:24.827 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-08-18T21:48:22.135 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:48:22.135 INFO:tasks.workunit.client.0.smithi099.stdout: "bytes_written": 1073741824, 2024-08-18T21:48:22.136 INFO:tasks.workunit.client.0.smithi099.stdout: "blocksize": 4194304, 2024-08-18T21:48:22.136 INFO:tasks.workunit.client.0.smithi099.stdout: "elapsed_sec": 27.811327697999999, 2024-08-18T21:48:22.136 INFO:tasks.workunit.client.0.smithi099.stdout: "bytes_per_sec": 38608074.941967487, 2024-08-18T21:48:22.136 INFO:tasks.workunit.client.0.smithi099.stdout: "iops": 9.2048823695105284 2024-08-18T21:48:22.136 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:48:22.162 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-08-18T21:48:22.635 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:48:22.635 INFO:tasks.workunit.client.0.smithi099.stdout: "bytes_written": 10, 2024-08-18T21:48:22.635 INFO:tasks.workunit.client.0.smithi099.stdout: "blocksize": 14456, 2024-08-18T21:48:22.635 INFO:tasks.workunit.client.0.smithi099.stdout: "elapsed_sec": 0.041807820000000002, 2024-08-18T21:48:22.635 INFO:tasks.workunit.client.0.smithi099.stdout: "bytes_per_sec": 239.18970183090147, 2024-08-18T21:48:22.635 INFO:tasks.workunit.client.0.smithi099.stdout: "iops": 0.016546050209663907 2024-08-18T21:48:22.635 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:48:22.667 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-08-18T21:48:22.985 INFO:tasks.workunit.client.0.smithi099.stdout:{ 2024-08-18T21:48:22.985 INFO:tasks.workunit.client.0.smithi099.stdout: "bytes_written": 1, 2024-08-18T21:48:22.985 INFO:tasks.workunit.client.0.smithi099.stdout: "blocksize": 14456, 2024-08-18T21:48:22.985 INFO:tasks.workunit.client.0.smithi099.stdout: "elapsed_sec": 0.040240400000000003, 2024-08-18T21:48:22.986 INFO:tasks.workunit.client.0.smithi099.stdout: "bytes_per_sec": 24.85064760787666, 2024-08-18T21:48:22.986 INFO:tasks.workunit.client.0.smithi099.stdout: "iops": 0.0017190542064109476 2024-08-18T21:48:22.986 INFO:tasks.workunit.client.0.smithi099.stdout:} 2024-08-18T21:48:23.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-08-18T21:48:23.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-08-18T21:48:23.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:48:23.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-08-18T21:48:23.014 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:48:23.014 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:48:23.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:48:23.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:48:23.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:48:23.325 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:48:23.325 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:48:23.326 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:48:23.327 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:48:23.328 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:48:23.328 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:48:23.329 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:48:23.330 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:48:23.330 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:48:23.331 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:48:23.331 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:48:23.332 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:48:23.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:48:23.334 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-08-18T21:48:23.373 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:48:23.373 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:48:23.373 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95377 2024-08-18T21:48:23.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95377 2024-08-18T21:48:23.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:48:23.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:48:23.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2331: main: code=0 2024-08-18T21:48:23.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: teardown td/osd-bench 0 2024-08-18T21:48:23.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-08-18T21:48:23.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-08-18T21:48:23.377 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-08-18T21:48:23.377 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:48:23.377 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:48:23.377 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:48:23.377 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:48:23.377 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:48:23.379 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:48:23.380 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:48:23.381 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:48:23.381 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:48:23.382 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:48:23.382 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:48:23.383 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:48:23.384 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:48:23.384 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:48:23.384 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:48:23.385 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:48:23.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:48:23.386 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-08-18T21:48:23.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-08-18T21:48:23.388 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:48:23.388 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:48:23.388 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95377 2024-08-18T21:48:23.388 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95377 2024-08-18T21:48:23.390 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:48:23.390 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:48:23.390 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: main: return 0 2024-08-18T21:48:23.391 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-08-18T21:48:23.391 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-18T21:48:23.443 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-08-18T21:48:23.444 DEBUG:teuthology.orchestra.run.smithi099:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-08-18T21:48:23.498 INFO:tasks.workunit.client.0.smithi099.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-08-18T21:48:23.505 INFO:tasks.workunit.client.0.smithi099.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-08-18T21:48:23.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-18T21:48:23.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-08-18T21:48:23.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export PYTHONWARNINGS=ignore 2024-08-18T21:48:23.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: PYTHONWARNINGS=ignore 2024-08-18T21:48:23.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: export CEPH_CONF=/dev/null 2024-08-18T21:48:23.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: CEPH_CONF=/dev/null 2024-08-18T21:48:23.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: unset CEPH_ARGS 2024-08-18T21:48:23.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: local code 2024-08-18T21:48:23.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: run td/osd-bluefs-volume-ops 2024-08-18T21:48:23.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:48:23.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-08-18T21:48:23.507 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-08-18T21:48:23.507 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-08-18T21:48:23.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-08-18T21:48:23.510 INFO:tasks.workunit.client.0.smithi099.stderr:TEST_bluestore2 2024-08-18T21:48:23.510 INFO:tasks.workunit.client.0.smithi099.stderr:TEST_bluestore_expand' 2024-08-18T21:48:23.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-08-18T21:48:23.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-08-18T21:48:23.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:48:23.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-08-18T21:48:23.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:48:23.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:48:23.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-18T21:48:23.511 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:48:23.511 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:48:23.511 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:48:23.511 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:48:23.511 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:48:23.512 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:48:23.512 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:48:23.513 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:48:23.514 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:48:23.515 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:48:23.515 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:48:23.515 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:48:23.516 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:48:23.516 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:48:23.517 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:48:23.517 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:48:23.517 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:48:23.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:48:23.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-18T21:48:23.520 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:48:23.520 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:48:23.521 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:48:23.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97620 2024-08-18T21:48:23.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:48:23.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:48:23.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-08-18T21:48:23.523 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-18T21:48:23.523 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:48:23.523 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:48:23.523 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.97620 2024-08-18T21:48:23.525 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-18T21:48:23.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-08-18T21:48:23.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-08-18T21:48:23.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-18T21:48:23.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-08-18T21:48:23.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-08-18T21:48:23.525 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:48:23.526 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-08-18T21:48:23.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-08-18T21:48:23.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-08-18T21:48:23.527 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-08-18T21:48:23.527 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-08-18T21:48:23.527 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-08-18T21:48:23.527 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-08-18T21:48:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none ' 2024-08-18T21:48:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-08-18T21:48:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-08-18T21:48:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-08-18T21:48:23.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-08-18T21:48:23.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-08-18T21:48:23.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-08-18T21:48:23.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-08-18T21:48:23.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-08-18T21:48:23.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:48:23.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-18T21:48:23.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-18T21:48:23.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-18T21:48:23.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-08-18T21:48:23.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-08-18T21:48:24.004 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-18T21:48:24.004 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:48:24.004 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:48:24.004 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:48:24.004 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:48:24.005 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:48:24.005 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:48:24.005 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-18T21:48:24.249 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-18T21:48:24.251 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-18T21:48:24.251 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:48:24.251 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:48:24.253 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-18T21:48:24.255 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-18T21:48:24.255 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:48:24.255 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:48:24.255 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:48:24.255 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:48:24.255 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:48:24.255 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:48:24.255 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97620/ceph-mon.a.asok 2024-08-18T21:48:24.255 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:48:24.255 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97620/ceph-mon.a.asok config get fsid 2024-08-18T21:48:24.368 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-18T21:48:24.368 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:48:24.368 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:48:24.368 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-18T21:48:24.369 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-18T21:48:24.369 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:48:24.369 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:48:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:48:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:48:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:48:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:48:24.370 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97620/ceph-mon.a.asok 2024-08-18T21:48:24.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:48:24.371 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97620/ceph-mon.a.asok config get mon_host 2024-08-18T21:48:24.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-08-18T21:48:24.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:48:24.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-18T21:48:24.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-18T21:48:24.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-18T21:48:24.479 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-08-18T21:48:24.480 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-18T21:48:25.415 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-18T21:48:25.415 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:48:25.415 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:48:25.415 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:48:25.415 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:48:25.416 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:48:25.416 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:48:25.416 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:48:25.417 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:48:25.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-08-18T21:48:25.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:48:25.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:48:25.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-18T21:48:25.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:48:25.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-18T21:48:25.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:48:25.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:48:25.484 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:48:25.485 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:48:25.486 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:48:25.486 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:48:25.486 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-18T21:48:25.486 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:48:25.486 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0070fe81-9b0d-489d-b9a5-ad08cec4b62e 2024-08-18T21:48:25.487 INFO:tasks.workunit.client.0.smithi099.stdout:add osd0 0070fe81-9b0d-489d-b9a5-ad08cec4b62e 2024-08-18T21:48:25.487 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0070fe81-9b0d-489d-b9a5-ad08cec4b62e' 2024-08-18T21:48:25.487 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:48:25.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApbMJmGTQ/HhAA0kAItR1iZw2G728gZbjVRA== 2024-08-18T21:48:25.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApbMJmGTQ/HhAA0kAItR1iZw2G728gZbjVRA=="}' 2024-08-18T21:48:25.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0070fe81-9b0d-489d-b9a5-ad08cec4b62e -i td/osd-bluefs-volume-ops/0/new.json 2024-08-18T21:48:25.793 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:48:25.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-08-18T21:48:25.809 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApbMJmGTQ/HhAA0kAItR1iZw2G728gZbjVRA== --osd-uuid 0070fe81-9b0d-489d-b9a5-ad08cec4b62e 2024-08-18T21:48:25.842 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:48:25.844+0000 7f0b1726c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:48:25.861 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:48:25.864+0000 7f0b1726c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:48:25.863 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:48:25.864+0000 7f0b1726c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:48:25.863 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:48:25.868+0000 7f0b1726c7c0 -1 bdev(0x55b080fd7c00 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-08-18T21:48:25.863 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:48:25.868+0000 7f0b1726c7c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-08-18T21:48:35.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-08-18T21:48:35.622 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:48:35.624 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd0 key to auth repository 2024-08-18T21:48:35.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-18T21:48:35.624 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:48:36.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-18T21:48:36.194 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:48:36.194 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:48:36.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:48:36.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:48:36.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:48:36.225 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:48:36.227+0000 7f7c58e3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:48:36.257 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:48:36.259+0000 7f7c58e3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:48:36.259 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:48:36.263+0000 7f7c58e3c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:48:36.630 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-18T21:48:36.631 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:48:36.631 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:48:36.631 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:48:36.631 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:48:36.631 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:36.631 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:48:36.631 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:48:36.632 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:36.632 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:37.027 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:38.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:38.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:38.029 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:48:38.029 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:48:38.029 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:38.029 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:38.220 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:48:38.223+0000 7f7c58e3c7c0 -1 Falling back to public interface 2024-08-18T21:48:38.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:39.452 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:39.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:39.453 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:48:39.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:48:39.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:39.454 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:39.857 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:40.858 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:40.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:40.859 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:48:40.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:48:40.860 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:40.860 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:41.282 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:42.284 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:42.284 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:42.284 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:48:42.285 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:48:42.285 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:42.285 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:42.713 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:43.395 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:48:43.399+0000 7f7c58e3c7c0 -1 osd.0 0 log_to_monitors true 2024-08-18T21:48:43.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:43.714 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:43.715 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:48:43.715 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:48:43.715 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:43.715 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:44.199 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:45.201 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:45.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:45.202 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:48:45.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:48:45.202 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:45.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:45.685 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:46.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:46.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:46.687 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:48:46.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:48:46.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:46.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:47.258 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:48.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:48.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:48.260 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:48:48.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:48:48.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:48.260 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:48.717 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:49.719 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:49.719 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:49.719 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:48:49.719 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:48:49.720 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:49.720 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:50.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:51.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:51.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:51.370 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:48:51.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:48:51.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:51.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:51.782 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:52.784 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:52.784 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:52.784 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:48:52.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:48:52.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:52.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:53.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:54.351 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:54.351 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:54.351 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:48:54.352 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:48:54.352 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:54.352 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:54.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:55.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:55.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:55.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:48:55.913 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:48:55.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:55.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:56.475 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:57.477 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:57.477 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:57.477 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:48:57.478 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:48:57.478 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:57.478 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:58.032 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:48:59.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:48:59.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:48:59.034 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:48:59.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:48:59.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:48:59.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:48:59.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:00.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:00.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:00.619 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:49:00.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:49:00.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:00.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:49:01.768 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:02.771 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:02.771 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:02.771 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:49:02.772 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:49:02.772 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:02.772 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:49:03.692 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:04.694 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:04.694 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:04.694 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:49:04.695 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:49:04.695 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:04.695 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:49:05.537 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:06.539 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:06.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:06.540 INFO:tasks.workunit.client.0.smithi099.stdout:19 2024-08-18T21:49:06.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-18T21:49:06.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:06.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:49:07.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:08.602 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:08.602 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:08.602 INFO:tasks.workunit.client.0.smithi099.stdout:20 2024-08-18T21:49:08.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-08-18T21:49:08.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:08.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:49:09.134 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:10.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:10.136 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:10.136 INFO:tasks.workunit.client.0.smithi099.stdout:21 2024-08-18T21:49:10.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-08-18T21:49:10.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:10.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:49:10.571 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1514144580,v1:127.0.0.1:6803/1514144580] [v2:127.0.0.1:6804/1514144580,v1:127.0.0.1:6805/1514144580] exists,up 0070fe81-9b0d-489d-b9a5-ad08cec4b62e 2024-08-18T21:49:10.571 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:49:10.572 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:49:10.572 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:49:10.572 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=99086 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:49:10.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-08-18T21:49:10.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-08-18T21:49:10.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:49:10.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:49:10.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:49:10.577 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:49:10.577 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:49:10.577 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:49:10.577 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:49:10.577 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:49:10.577 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:49:10.577 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:49:10.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:49:10.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:49:10.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:49:10.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:49:10.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:49:10.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:49:10.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:49:10.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:49:10.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:49:10.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:49:10.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:49:10.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:49:10.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-08-18T21:49:10.579 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:49:10.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=000c933c-86aa-4ddf-9c31-889e86e7877b 2024-08-18T21:49:10.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 000c933c-86aa-4ddf-9c31-889e86e7877b' 2024-08-18T21:49:10.580 INFO:tasks.workunit.client.0.smithi099.stdout:add osd1 000c933c-86aa-4ddf-9c31-889e86e7877b 2024-08-18T21:49:10.580 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:49:10.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBWbMJmJc8NJBAAsbCCtKujri+Q+HzR56uXfQ== 2024-08-18T21:49:10.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBWbMJmJc8NJBAAsbCCtKujri+Q+HzR56uXfQ=="}' 2024-08-18T21:49:10.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 000c933c-86aa-4ddf-9c31-889e86e7877b -i td/osd-bluefs-volume-ops/1/new.json 2024-08-18T21:49:11.080 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:49:11.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-08-18T21:49:11.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWbMJmJc8NJBAAsbCCtKujri+Q+HzR56uXfQ== --osd-uuid 000c933c-86aa-4ddf-9c31-889e86e7877b 2024-08-18T21:49:11.130 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:11.131+0000 7f813de327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:49:11.169 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:11.171+0000 7f813de327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:49:11.170 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:11.171+0000 7f813de327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:49:11.171 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:11.175+0000 7f813de327c0 -1 bdev(0x5586f6eb1c00 td/osd-bluefs-volume-ops/1/block) open stat got: (1) Operation not permitted 2024-08-18T21:49:11.171 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:11.175+0000 7f813de327c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-08-18T21:49:21.588 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-08-18T21:49:21.588 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:49:21.589 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd1 key to auth repository 2024-08-18T21:49:21.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-08-18T21:49:21.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:49:22.163 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:49:22.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-08-18T21:49:22.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:49:22.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:49:22.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:49:22.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:49:22.196 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:22.198+0000 7f655e13e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:49:22.211 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:22.214+0000 7f655e13e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:49:22.213 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:22.214+0000 7f655e13e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:49:22.632 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-08-18T21:49:22.632 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:49:22.632 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:49:22.632 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:49:22.632 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:49:22.632 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:22.632 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:49:22.632 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:49:22.633 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:22.633 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:23.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:23.966 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:23.966+0000 7f655e13e7c0 -1 Falling back to public interface 2024-08-18T21:49:24.101 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:24.101 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:24.102 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:49:24.102 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:49:24.102 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:24.102 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:24.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:25.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:25.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:25.535 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:49:25.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:49:25.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:25.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:25.955 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:26.957 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:26.958 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:26.958 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:49:26.958 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:49:26.958 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:26.958 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:27.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:28.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:28.374 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:49:28.374 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:28.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:49:28.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:28.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:28.865 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:29.199 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:29.202+0000 7f655e13e7c0 -1 osd.1 0 log_to_monitors true 2024-08-18T21:49:29.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:29.867 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:29.867 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:49:29.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:49:29.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:29.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:30.595 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:31.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:31.596 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:31.596 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:49:31.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:49:31.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:31.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:32.145 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:33.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:33.147 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:33.147 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:49:33.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:49:33.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:33.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:33.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:34.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:34.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:34.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:49:34.567 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:49:34.567 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:34.567 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:35.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:36.036 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:36.037 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:36.037 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:49:36.037 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:49:36.037 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:36.037 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:36.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:37.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:37.483 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:37.483 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:49:37.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:49:37.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:37.484 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:37.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:38.968 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:38.969 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:49:38.969 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:38.969 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:49:38.969 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:38.969 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:39.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:40.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:40.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:40.387 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:49:40.388 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:49:40.388 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:40.388 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:40.957 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:41.959 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:41.959 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:41.959 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:49:41.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:49:41.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:41.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:42.511 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:43.513 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:43.513 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:43.513 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:49:43.514 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:49:43.514 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:43.514 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:44.054 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:45.056 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:45.056 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:45.057 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:49:45.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:49:45.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:45.057 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:45.715 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:46.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:46.716 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:46.716 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:49:46.717 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:49:46.717 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:46.717 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:47.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:48.523 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:48.523 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:48.523 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:49:48.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:49:48.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:48.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:49.409 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:50.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:50.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:50.411 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:49:50.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:49:50.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:50.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:51.259 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:52.261 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:52.261 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:52.261 INFO:tasks.workunit.client.0.smithi099.stdout:19 2024-08-18T21:49:52.261 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-18T21:49:52.262 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:52.262 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:52.771 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:53.772 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:53.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:53.773 INFO:tasks.workunit.client.0.smithi099.stdout:20 2024-08-18T21:49:53.773 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-08-18T21:49:53.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:53.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:54.582 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:49:55.584 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:49:55.584 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:49:55.584 INFO:tasks.workunit.client.0.smithi099.stdout:21 2024-08-18T21:49:55.585 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-08-18T21:49:55.585 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:49:55.585 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:49:56.006 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/985670901,v1:127.0.0.1:6811/985670901] [v2:127.0.0.1:6812/985670901,v1:127.0.0.1:6813/985670901] exists,up 000c933c-86aa-4ddf-9c31-889e86e7877b 2024-08-18T21:49:56.006 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:49:56.006 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:49:56.006 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:49:56.007 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-08-18T21:49:56.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=101692 2024-08-18T21:49:56.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-08-18T21:49:56.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:49:56.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:49:56.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-08-18T21:49:56.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:49:56.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-08-18T21:49:56.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:49:56.010 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:49:56.011 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:49:56.011 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:49:56.011 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:49:56.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:49:56.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-08-18T21:49:56.014 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:49:56.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dab4835d-5f3e-4530-8813-db3b51c8e6fa 2024-08-18T21:49:56.014 INFO:tasks.workunit.client.0.smithi099.stdout:add osd2 dab4835d-5f3e-4530-8813-db3b51c8e6fa 2024-08-18T21:49:56.014 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 dab4835d-5f3e-4530-8813-db3b51c8e6fa' 2024-08-18T21:49:56.014 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:49:56.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEbMJmbqRLAhAAyd6kyE/NsdN1nnyRhY7cCw== 2024-08-18T21:49:56.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEbMJmbqRLAhAAyd6kyE/NsdN1nnyRhY7cCw=="}' 2024-08-18T21:49:56.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dab4835d-5f3e-4530-8813-db3b51c8e6fa -i td/osd-bluefs-volume-ops/2/new.json 2024-08-18T21:49:56.662 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:49:56.680 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-08-18T21:49:56.681 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEbMJmbqRLAhAAyd6kyE/NsdN1nnyRhY7cCw== --osd-uuid dab4835d-5f3e-4530-8813-db3b51c8e6fa 2024-08-18T21:49:56.716 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:56.717+0000 7f2d9f9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:49:56.725 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:56.725+0000 7f2d9f9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:49:56.726 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:56.725+0000 7f2d9f9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:49:56.727 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:56.725+0000 7f2d9f9437c0 -1 bdev(0x56237b1ca800 td/osd-bluefs-volume-ops/2/block) open stat got: (1) Operation not permitted 2024-08-18T21:49:56.727 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:49:56.725+0000 7f2d9f9437c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-08-18T21:50:07.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-08-18T21:50:07.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:50:07.534 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd2 key to auth repository 2024-08-18T21:50:07.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-08-18T21:50:07.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:50:08.105 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:50:08.105 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-08-18T21:50:08.106 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:50:08.106 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:50:08.106 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:50:08.106 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:50:08.137 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:08.141+0000 7f21f81ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:50:08.180 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:08.185+0000 7f21f81ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:50:08.182 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:08.185+0000 7f21f81ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:50:08.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-08-18T21:50:08.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:50:08.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:50:08.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:50:08.550 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:50:08.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:08.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:50:08.551 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:50:08.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:08.551 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:08.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:09.987 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:09.987 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:09.987 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:50:09.988 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:50:09.988 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:09.988 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:10.166 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:10.169+0000 7f21f81ab7c0 -1 Falling back to public interface 2024-08-18T21:50:10.441 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:11.443 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:11.443 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:11.443 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:50:11.444 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:50:11.444 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:11.444 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:11.903 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:12.904 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:12.904 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:12.905 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:50:12.905 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:50:12.905 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:12.905 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:13.326 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:14.327 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:14.327 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:14.328 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:50:14.328 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:50:14.328 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:14.328 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:14.777 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:15.487 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:15.489+0000 7f21f81ab7c0 -1 osd.2 0 log_to_monitors true 2024-08-18T21:50:15.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:15.779 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:15.780 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:50:15.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:50:15.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:15.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:16.317 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:17.318 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:17.319 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:50:17.319 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:17.320 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:50:17.320 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:17.320 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:17.784 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:18.786 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:18.786 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:18.786 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:50:18.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:50:18.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:18.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:19.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:20.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:20.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:20.491 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:50:20.492 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:50:20.492 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:20.492 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:21.032 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:22.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:22.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:22.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:50:22.034 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:50:22.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:22.034 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:22.830 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:23.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:23.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:23.832 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:50:23.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:50:23.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:23.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:24.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:25.531 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:25.531 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:25.531 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:50:25.532 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:50:25.532 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:25.532 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:25.974 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:26.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:26.976 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:26.976 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:50:26.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:50:26.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:26.977 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:27.510 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:28.512 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:28.512 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:28.512 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:50:28.512 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:50:28.513 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:28.513 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:29.021 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:30.023 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:30.023 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:30.024 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:50:30.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:50:30.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:30.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:31.010 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:32.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:32.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:32.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:50:32.013 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:50:32.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:32.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:33.446 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:34.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:34.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:34.448 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:50:34.449 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:50:34.449 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:34.449 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:35.330 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:36.331 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:36.332 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:36.332 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:50:36.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:50:36.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:36.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:37.411 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:38.412 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:38.413 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:38.413 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:50:38.413 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:50:38.413 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:38.414 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:39.524 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:40.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:40.526 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:40.526 INFO:tasks.workunit.client.0.smithi099.stdout:19 2024-08-18T21:50:40.527 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-18T21:50:40.527 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:40.527 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:40.963 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:41.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:41.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:41.965 INFO:tasks.workunit.client.0.smithi099.stdout:20 2024-08-18T21:50:41.965 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-08-18T21:50:41.965 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:41.965 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:50:42.388 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4015540638,v1:127.0.0.1:6819/4015540638] [v2:127.0.0.1:6820/4015540638,v1:127.0.0.1:6821/4015540638] exists,up dab4835d-5f3e-4530-8813-db3b51c8e6fa 2024-08-18T21:50:42.389 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:50:42.389 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:50:42.389 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:50:42.389 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-08-18T21:50:42.392 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=104296 2024-08-18T21:50:42.392 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-08-18T21:50:42.392 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:50:42.392 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:50:42.392 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-08-18T21:50:42.392 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:50:42.393 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:50:42.394 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:50:42.394 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:50:42.394 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:50:42.395 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-08-18T21:50:42.396 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:50:42.397 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fc8c4dc2-5d7e-447d-b7fe-0621f00c25c1 2024-08-18T21:50:42.397 INFO:tasks.workunit.client.0.smithi099.stdout:add osd3 fc8c4dc2-5d7e-447d-b7fe-0621f00c25c1 2024-08-18T21:50:42.397 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 fc8c4dc2-5d7e-447d-b7fe-0621f00c25c1' 2024-08-18T21:50:42.397 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:50:42.418 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCybMJmEqklGRAAJLNTV+YNPi3NeYwM9pM0/A== 2024-08-18T21:50:42.418 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCybMJmEqklGRAAJLNTV+YNPi3NeYwM9pM0/A=="}' 2024-08-18T21:50:42.418 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fc8c4dc2-5d7e-447d-b7fe-0621f00c25c1 -i td/osd-bluefs-volume-ops/3/new.json 2024-08-18T21:50:42.938 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:50:42.953 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-08-18T21:50:42.955 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCybMJmEqklGRAAJLNTV+YNPi3NeYwM9pM0/A== --osd-uuid fc8c4dc2-5d7e-447d-b7fe-0621f00c25c1 2024-08-18T21:50:42.987 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:42.988+0000 7f919fa687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:50:43.020 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:43.024+0000 7f919fa687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:50:43.021 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:43.024+0000 7f919fa687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:50:43.022 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:43.024+0000 7f919fa687c0 -1 bdev(0x561289ffbc00 td/osd-bluefs-volume-ops/3/block) open stat got: (1) Operation not permitted 2024-08-18T21:50:43.022 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:43.024+0000 7f919fa687c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-08-18T21:50:53.585 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-08-18T21:50:53.586 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:50:53.586 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd3 key to auth repository 2024-08-18T21:50:53.587 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-08-18T21:50:53.587 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:50:54.148 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.3 2024-08-18T21:50:54.149 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-08-18T21:50:54.149 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:50:54.149 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:50:54.149 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:50:54.149 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:50:54.173 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:54.175+0000 7fea45f7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:50:54.175 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:54.175+0000 7fea45f7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:50:54.177 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:54.179+0000 7fea45f7d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:50:54.626 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-08-18T21:50:54.626 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:50:54.626 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-18T21:50:54.626 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:50:54.626 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:50:54.626 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:54.627 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:50:54.627 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:50:54.627 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:54.627 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:50:55.164 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:55.727 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:50:55.727+0000 7fea45f7d7c0 -1 Falling back to public interface 2024-08-18T21:50:56.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:56.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:56.166 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:50:56.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:50:56.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:56.167 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:50:56.588 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:57.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:57.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:57.590 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:50:57.591 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:50:57.591 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:57.591 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:50:58.025 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:50:59.027 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:50:59.027 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:50:59.028 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:50:59.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:50:59.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:50:59.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:50:59.443 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:00.444 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:00.445 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:00.445 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:51:00.445 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:51:00.446 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:00.446 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:00.918 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:01.083 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:51:01.083+0000 7fea45f7d7c0 -1 osd.3 0 log_to_monitors true 2024-08-18T21:51:01.920 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:01.920 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:01.920 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:51:01.921 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:51:01.921 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:01.921 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:02.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:03.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:03.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:03.346 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:51:03.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:51:03.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:03.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:03.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:04.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:04.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:04.805 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T21:51:04.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T21:51:04.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:04.805 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:05.338 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:06.340 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:06.340 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:06.341 INFO:tasks.workunit.client.0.smithi099.stdout:8 2024-08-18T21:51:06.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-08-18T21:51:06.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:06.341 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:06.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:07.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:07.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:07.889 INFO:tasks.workunit.client.0.smithi099.stdout:9 2024-08-18T21:51:07.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-08-18T21:51:07.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:07.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:08.405 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:09.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:09.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:09.407 INFO:tasks.workunit.client.0.smithi099.stdout:10 2024-08-18T21:51:09.407 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-08-18T21:51:09.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:09.408 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:09.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:10.965 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:10.965 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:10.966 INFO:tasks.workunit.client.0.smithi099.stdout:11 2024-08-18T21:51:10.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-08-18T21:51:10.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:10.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:11.538 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:12.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:12.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:12.541 INFO:tasks.workunit.client.0.smithi099.stdout:12 2024-08-18T21:51:12.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-08-18T21:51:12.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:12.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:13.272 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:14.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:14.274 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:14.274 INFO:tasks.workunit.client.0.smithi099.stdout:13 2024-08-18T21:51:14.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-08-18T21:51:14.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:14.275 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:14.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:15.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:15.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:15.797 INFO:tasks.workunit.client.0.smithi099.stdout:14 2024-08-18T21:51:15.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-08-18T21:51:15.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:15.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:16.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:17.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:17.689 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:17.689 INFO:tasks.workunit.client.0.smithi099.stdout:15 2024-08-18T21:51:17.690 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-08-18T21:51:17.690 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:17.690 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:18.383 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:19.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:19.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:19.385 INFO:tasks.workunit.client.0.smithi099.stdout:16 2024-08-18T21:51:19.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-08-18T21:51:19.386 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:19.386 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:20.127 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:21.129 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:21.129 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:21.129 INFO:tasks.workunit.client.0.smithi099.stdout:17 2024-08-18T21:51:21.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-08-18T21:51:21.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:21.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:21.559 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:22.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:22.560 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:22.560 INFO:tasks.workunit.client.0.smithi099.stdout:18 2024-08-18T21:51:22.561 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-08-18T21:51:22.561 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:22.561 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:23.628 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:24.629 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:24.630 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:24.630 INFO:tasks.workunit.client.0.smithi099.stdout:19 2024-08-18T21:51:24.630 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-08-18T21:51:24.631 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:24.631 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:25.403 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:51:26.405 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:51:26.405 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:51:26.405 INFO:tasks.workunit.client.0.smithi099.stdout:20 2024-08-18T21:51:26.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-08-18T21:51:26.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:51:26.406 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:51:26.876 INFO:tasks.workunit.client.0.smithi099.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4209737924,v1:127.0.0.1:6827/4209737924] [v2:127.0.0.1:6828/4209737924,v1:127.0.0.1:6829/4209737924] exists,up fc8c4dc2-5d7e-447d-b7fe-0621f00c25c1 2024-08-18T21:51:26.877 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:51:26.877 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:51:26.877 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:51:26.878 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-08-18T21:51:26.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=106875 2024-08-18T21:51:26.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-08-18T21:51:31.881 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-08-18T21:51:31.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-08-18T21:51:32.421 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'foo' already exists 2024-08-18T21:51:32.437 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-18T21:51:33.439 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-08-18T21:51:33.499 INFO:tasks.workunit.client.0.smithi099.stdout:hints = 1 2024-08-18T21:51:33.500 INFO:tasks.workunit.client.0.smithi099.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-08-18T21:51:33.500 INFO:tasks.workunit.client.0.smithi099.stdout:Object prefix: benchmark_data_smithi099_108265 2024-08-18T21:51:33.500 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T21:51:33.500 INFO:tasks.workunit.client.0.smithi099.stdout: 0 0 0 0 0 0 - 0 2024-08-18T21:51:34.500 INFO:tasks.workunit.client.0.smithi099.stdout: 1 16 42 26 0.101556 0.101562 0.332684 0.533724 2024-08-18T21:51:35.500 INFO:tasks.workunit.client.0.smithi099.stdout: 2 16 93 77 0.150378 0.199219 0.260735 0.385104 2024-08-18T21:51:36.500 INFO:tasks.workunit.client.0.smithi099.stdout: 3 16 142 126 0.164045 0.191406 0.308614 0.358032 2024-08-18T21:51:37.500 INFO:tasks.workunit.client.0.smithi099.stdout: 4 16 201 185 0.180642 0.230469 0.283743 0.338161 2024-08-18T21:51:38.500 INFO:tasks.workunit.client.0.smithi099.stdout: 5 16 250 234 0.182791 0.191406 0.30076 0.331015 2024-08-18T21:51:39.501 INFO:tasks.workunit.client.0.smithi099.stdout: 6 16 306 290 0.18878 0.21875 0.326628 0.323382 2024-08-18T21:51:40.501 INFO:tasks.workunit.client.0.smithi099.stdout: 7 16 354 338 0.188595 0.1875 0.317205 0.321475 2024-08-18T21:51:41.501 INFO:tasks.workunit.client.0.smithi099.stdout: 8 16 410 394 0.19236 0.21875 0.31089 0.317985 2024-08-18T21:51:42.501 INFO:tasks.workunit.client.0.smithi099.stdout: 9 16 466 450 0.195289 0.21875 0.317808 0.31667 2024-08-18T21:51:43.501 INFO:tasks.workunit.client.0.smithi099.stdout: 10 16 514 498 0.194508 0.1875 0.288647 0.31608 2024-08-18T21:51:44.501 INFO:tasks.workunit.client.0.smithi099.stdout: 11 16 562 546 0.193868 0.1875 0.357114 0.316564 2024-08-18T21:51:45.502 INFO:tasks.workunit.client.0.smithi099.stdout: 12 16 618 602 0.19594 0.21875 0.322234 0.314653 2024-08-18T21:51:46.502 INFO:tasks.workunit.client.0.smithi099.stdout: 13 16 668 652 0.195888 0.195312 0.283456 0.31418 2024-08-18T21:51:47.502 INFO:tasks.workunit.client.0.smithi099.stdout: 14 16 715 699 0.195007 0.183594 0.271354 0.314983 2024-08-18T21:51:48.502 INFO:tasks.workunit.client.0.smithi099.stdout: 15 16 773 757 0.19711 0.226562 0.288521 0.315403 2024-08-18T21:51:49.502 INFO:tasks.workunit.client.0.smithi099.stdout: 16 16 822 806 0.196751 0.191406 0.311472 0.314452 2024-08-18T21:51:50.502 INFO:tasks.workunit.client.0.smithi099.stdout: 17 16 875 859 0.197355 0.207031 0.341874 0.313929 2024-08-18T21:51:51.503 INFO:tasks.workunit.client.0.smithi099.stdout: 18 16 923 907 0.196805 0.1875 0.300667 0.313791 2024-08-18T21:51:52.503 INFO:tasks.workunit.client.0.smithi099.stdout: 19 16 955 939 0.193025 0.125 0.697914 0.320134 2024-08-18T21:51:53.503 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:51:53.507518+0000 min lat: 0.213386 max lat: 0.833468 avg lat: 0.318813 2024-08-18T21:51:53.503 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T21:51:53.503 INFO:tasks.workunit.client.0.smithi099.stdout: 20 16 1013 997 0.1947 0.226562 0.275359 0.318813 2024-08-18T21:51:54.503 INFO:tasks.workunit.client.0.smithi099.stdout: 21 16 1067 1051 0.195471 0.210938 0.292319 0.317696 2024-08-18T21:51:55.503 INFO:tasks.workunit.client.0.smithi099.stdout: 22 16 1119 1103 0.195818 0.203125 0.298376 0.317186 2024-08-18T21:51:56.504 INFO:tasks.workunit.client.0.smithi099.stdout: 23 16 1168 1152 0.195625 0.191406 0.309839 0.317534 2024-08-18T21:51:57.504 INFO:tasks.workunit.client.0.smithi099.stdout: 24 16 1218 1202 0.195611 0.195312 0.282362 0.317349 2024-08-18T21:51:58.504 INFO:tasks.workunit.client.0.smithi099.stdout: 25 16 1273 1257 0.196379 0.214844 0.326412 0.316918 2024-08-18T21:51:59.504 INFO:tasks.workunit.client.0.smithi099.stdout: 26 16 1322 1306 0.196187 0.191406 0.283681 0.316315 2024-08-18T21:52:00.504 INFO:tasks.workunit.client.0.smithi099.stdout: 27 16 1376 1360 0.196733 0.210938 0.333328 0.316095 2024-08-18T21:52:01.504 INFO:tasks.workunit.client.0.smithi099.stdout: 28 16 1424 1408 0.196402 0.1875 0.313755 0.315823 2024-08-18T21:52:02.504 INFO:tasks.workunit.client.0.smithi099.stdout: 29 16 1482 1466 0.197441 0.226562 0.300215 0.314992 2024-08-18T21:52:03.504 INFO:tasks.workunit.client.0.smithi099.stdout: 30 16 1530 1514 0.197109 0.1875 0.290505 0.314982 2024-08-18T21:52:04.505 INFO:tasks.workunit.client.0.smithi099.stdout:Total time run: 30.1383 2024-08-18T21:52:04.505 INFO:tasks.workunit.client.0.smithi099.stdout:Total writes made: 1530 2024-08-18T21:52:04.505 INFO:tasks.workunit.client.0.smithi099.stdout:Write size: 4096 2024-08-18T21:52:04.505 INFO:tasks.workunit.client.0.smithi099.stdout:Object size: 4096 2024-08-18T21:52:04.505 INFO:tasks.workunit.client.0.smithi099.stdout:Bandwidth (MB/sec): 0.198305 2024-08-18T21:52:04.505 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Bandwidth: 0.027329 2024-08-18T21:52:04.505 INFO:tasks.workunit.client.0.smithi099.stdout:Max bandwidth (MB/sec): 0.230469 2024-08-18T21:52:04.505 INFO:tasks.workunit.client.0.smithi099.stdout:Min bandwidth (MB/sec): 0.101562 2024-08-18T21:52:04.505 INFO:tasks.workunit.client.0.smithi099.stdout:Average IOPS: 50 2024-08-18T21:52:04.505 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev IOPS: 6.99622 2024-08-18T21:52:04.506 INFO:tasks.workunit.client.0.smithi099.stdout:Max IOPS: 59 2024-08-18T21:52:04.506 INFO:tasks.workunit.client.0.smithi099.stdout:Min IOPS: 26 2024-08-18T21:52:04.506 INFO:tasks.workunit.client.0.smithi099.stdout:Average Latency(s): 0.314614 2024-08-18T21:52:04.506 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Latency(s): 0.0625544 2024-08-18T21:52:04.506 INFO:tasks.workunit.client.0.smithi099.stdout:Max latency(s): 0.833468 2024-08-18T21:52:04.506 INFO:tasks.workunit.client.0.smithi099.stdout:Min latency(s): 0.213386 2024-08-18T21:52:04.653 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-08-18T21:52:04.653 INFO:tasks.workunit.client.0.smithi099.stdout:after bench 2024-08-18T21:52:04.653 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 99086 2024-08-18T21:52:04.653 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-08-18T21:52:05.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 99086 2024-08-18T21:52:05.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (99086) - No such process 2024-08-18T21:52:05.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-08-18T21:52:06.140 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already down. 2024-08-18T21:52:06.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 101692 2024-08-18T21:52:06.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-08-18T21:52:07.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 101692 2024-08-18T21:52:07.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (101692) - No such process 2024-08-18T21:52:07.158 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-08-18T21:52:07.591 INFO:tasks.workunit.client.0.smithi099.stderr:osd.1 is already down. 2024-08-18T21:52:07.607 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 104296 2024-08-18T21:52:07.607 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-08-18T21:52:08.609 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 104296 2024-08-18T21:52:08.609 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (104296) - No such process 2024-08-18T21:52:08.609 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-08-18T21:52:09.102 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already down. 2024-08-18T21:52:09.120 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 106875 2024-08-18T21:52:09.120 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-08-18T21:52:10.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 106875 2024-08-18T21:52:10.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (106875) - No such process 2024-08-18T21:52:10.123 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-08-18T21:52:10.712 INFO:tasks.workunit.client.0.smithi099.stderr:osd.3 is already down. 2024-08-18T21:52:10.729 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-18T21:52:13.105 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:52:13.130 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-18T21:52:15.505 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:52:15.529 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-18T21:52:17.901 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:52:17.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-18T21:52:20.289 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:52:20.313 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-08-18T21:52:20.330 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-08-18T21:52:20.358 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:52:21.554 INFO:tasks.workunit.client.0.smithi099.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-08-18T21:52:21.554 INFO:tasks.workunit.client.0.smithi099.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-08-18T21:52:21.554 INFO:tasks.workunit.client.0.smithi099.stdout:2 : device size 0x100000000 : using 0x804d5000(2.0 GiB) 2024-08-18T21:52:21.554 INFO:tasks.workunit.client.0.smithi099.stdout:Expanding DB/WAL... 2024-08-18T21:52:21.555 INFO:tasks.workunit.client.0.smithi099.stdout:0 : expanding to 0x536870912 2024-08-18T21:52:21.569 INFO:tasks.workunit.client.0.smithi099.stdout:0 : size label updated to 536870912 2024-08-18T21:52:21.569 INFO:tasks.workunit.client.0.smithi099.stdout:1 : expanding to 0x1073741824 2024-08-18T21:52:21.577 INFO:tasks.workunit.client.0.smithi099.stdout:1 : size label updated to 1073741824 2024-08-18T21:52:21.577 INFO:tasks.workunit.client.0.smithi099.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-08-18T21:52:21.727 INFO:tasks.workunit.client.0.smithi099.stdout:2 : size label updated to 4294967296 2024-08-18T21:52:27.108 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-08-18T21:52:27.109 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-08-18T21:52:27.136 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:52:28.375 INFO:tasks.workunit.client.0.smithi099.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-08-18T21:52:28.375 INFO:tasks.workunit.client.0.smithi099.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-08-18T21:52:28.376 INFO:tasks.workunit.client.0.smithi099.stdout:2 : device size 0x101000000 : using 0x814a1000(2.0 GiB) 2024-08-18T21:52:28.376 INFO:tasks.workunit.client.0.smithi099.stdout:Expanding DB/WAL... 2024-08-18T21:52:28.376 INFO:tasks.workunit.client.0.smithi099.stdout:0 : expanding to 0x536870912 2024-08-18T21:52:28.389 INFO:tasks.workunit.client.0.smithi099.stdout:0 : size label updated to 536870912 2024-08-18T21:52:28.389 INFO:tasks.workunit.client.0.smithi099.stdout:1 : expanding to 0x1073741824 2024-08-18T21:52:28.398 INFO:tasks.workunit.client.0.smithi099.stdout:1 : size label updated to 1073741824 2024-08-18T21:52:28.398 INFO:tasks.workunit.client.0.smithi099.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-08-18T21:52:28.556 INFO:tasks.workunit.client.0.smithi099.stdout:2 : size label updated to 4311744512 2024-08-18T21:52:34.023 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-08-18T21:52:34.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-08-18T21:52:34.051 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:52:35.266 INFO:tasks.workunit.client.0.smithi099.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-08-18T21:52:35.266 INFO:tasks.workunit.client.0.smithi099.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-08-18T21:52:35.266 INFO:tasks.workunit.client.0.smithi099.stdout:2 : device size 0x100020000 : using 0x804ef000(2.0 GiB) 2024-08-18T21:52:35.266 INFO:tasks.workunit.client.0.smithi099.stdout:Expanding DB/WAL... 2024-08-18T21:52:35.266 INFO:tasks.workunit.client.0.smithi099.stdout:0 : expanding to 0x536870912 2024-08-18T21:52:35.279 INFO:tasks.workunit.client.0.smithi099.stdout:0 : size label updated to 536870912 2024-08-18T21:52:35.279 INFO:tasks.workunit.client.0.smithi099.stdout:1 : expanding to 0x1073741824 2024-08-18T21:52:35.288 INFO:tasks.workunit.client.0.smithi099.stdout:1 : size label updated to 1073741824 2024-08-18T21:52:35.288 INFO:tasks.workunit.client.0.smithi099.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-08-18T21:52:35.429 INFO:tasks.workunit.client.0.smithi099.stdout:2 : size label updated to 4295098368 2024-08-18T21:52:41.099 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-08-18T21:52:41.100 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-08-18T21:52:41.128 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:52:42.370 INFO:tasks.workunit.client.0.smithi099.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-08-18T21:52:42.371 INFO:tasks.workunit.client.0.smithi099.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-08-18T21:52:42.371 INFO:tasks.workunit.client.0.smithi099.stdout:2 : device size 0xfff00000 : using 0x803b6000(2.0 GiB) 2024-08-18T21:52:42.371 INFO:tasks.workunit.client.0.smithi099.stdout:Expanding DB/WAL... 2024-08-18T21:52:42.371 INFO:tasks.workunit.client.0.smithi099.stdout:0 : expanding to 0x536870912 2024-08-18T21:52:42.387 INFO:tasks.workunit.client.0.smithi099.stdout:0 : size label updated to 536870912 2024-08-18T21:52:42.387 INFO:tasks.workunit.client.0.smithi099.stdout:1 : expanding to 0x1073741824 2024-08-18T21:52:42.395 INFO:tasks.workunit.client.0.smithi099.stdout:1 : size label updated to 1073741824 2024-08-18T21:52:42.396 INFO:tasks.workunit.client.0.smithi099.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-08-18T21:52:42.545 INFO:tasks.workunit.client.0.smithi099.stdout:2 : size label updated to 4293918720 2024-08-18T21:52:48.195 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-18T21:52:50.574 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:52:50.590 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-18T21:52:52.966 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:52:52.989 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-18T21:52:55.358 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:52:55.380 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-18T21:52:57.758 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:52:57.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-08-18T21:52:57.809 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:52:59.875 INFO:tasks.workunit.client.0.smithi099.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-08-18T21:52:59.876 INFO:tasks.workunit.client.0.smithi099.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-08-18T21:52:59.876 INFO:tasks.workunit.client.0.smithi099.stdout:2 : device size 0x100000000 : using 0x4d5000(4.8 MiB) 2024-08-18T21:52:59.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-08-18T21:52:59.906 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:53:02.822 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-08-18T21:53:02.830 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-18T21:53:04.710 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:53:04.732 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-08-18T21:53:04.758 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:53:07.674 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-08-18T21:53:07.678 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-18T21:53:09.566 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:53:09.585 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-08-18T21:53:09.612 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:53:12.518 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-08-18T21:53:12.518 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-08-18T21:53:12.528 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-18T21:53:13.898 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:53:13.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-08-18T21:53:13.940 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:53:14.798 INFO:tasks.workunit.client.0.smithi099.stderr:Migrate to WAL device isn't supported. 2024-08-18T21:53:14.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-08-18T21:53:14.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-18T21:53:17.178 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:53:17.198 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-08-18T21:53:17.224 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:53:20.134 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-08-18T21:53:20.144 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-18T21:53:22.022 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:53:22.041 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-08-18T21:53:22.068 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:53:24.216 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-18T21:53:26.090 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:53:26.110 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-08-18T21:53:26.110 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:53:26.110 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:53:26.110 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-18T21:53:26.110 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:53:26.110 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-18T21:53:26.110 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:53:26.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:53:26.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:53:26.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:53:26.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-18T21:53:26.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-18T21:53:26.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:53:26.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:53:26.111 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:53:26.113 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:53:26.113 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:53:26.113 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:53:26.113 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:53:26.113 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:53:26.114 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:53:26.114 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:53:26.114 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:53:26.114 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:53:26.114 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:53:26.114 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:53:26.114 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:53:26.114 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:53:26.114 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:53:26.115 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:53:26.115 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:53:26.115 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:53:26.115 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-18T21:53:26.115 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:53:26.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-18T21:53:26.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:53:26.116 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-08-18T21:53:26.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-18T21:53:26.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:53:26.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:53:26.117 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:53:26.147 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:26.146+0000 7f91032b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:26.150 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:26.150+0000 7f91032b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:26.151 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:26.150+0000 7f91032b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:26.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-18T21:53:26.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:53:26.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:53:26.604 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:53:26.604 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:53:26.604 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:53:26.604 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:26.605 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:53:26.605 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:26.605 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:53:27.041 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:28.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:28.043 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:28.043 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:53:28.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:53:28.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:28.044 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:53:28.132 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:28.130+0000 7f91032b17c0 -1 Falling back to public interface 2024-08-18T21:53:28.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:29.455 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:29.455 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:29.455 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:53:29.455 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:53:29.456 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:29.457 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:53:29.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:30.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:30.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:30.889 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:53:30.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:53:30.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:30.890 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:53:31.291 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:32.292 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:32.293 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:32.293 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:53:32.293 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:53:32.294 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:32.294 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:53:32.323 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:32.322+0000 7f91032b17c0 -1 osd.0 27 log_to_monitors true 2024-08-18T21:53:32.776 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 37 up_thru 22 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/1099571341,v1:127.0.0.1:6803/1099571341] [v2:127.0.0.1:6804/1099571341,v1:127.0.0.1:6805/1099571341] exists,up 0070fe81-9b0d-489d-b9a5-ad08cec4b62e 2024-08-18T21:53:32.777 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:53:32.777 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:53:32.777 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:53:32.777 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-18T21:53:32.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=120408 2024-08-18T21:53:32.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-08-18T21:53:32.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:53:32.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:53:32.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-18T21:53:32.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:53:32.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-08-18T21:53:32.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:53:32.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:53:32.780 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:53:32.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:53:32.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-08-18T21:53:32.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-08-18T21:53:32.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:53:32.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:53:32.781 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:53:32.781 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:53:32.781 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:53:32.781 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:53:32.783 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:53:32.783 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:53:32.783 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:53:32.783 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:53:32.783 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:53:32.783 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:53:32.783 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:53:32.783 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:53:32.783 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:53:32.783 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:53:32.783 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:53:32.784 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:53:32.784 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:53:32.784 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:53:32.784 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-08-18T21:53:32.784 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-18T21:53:32.784 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:53:32.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:53:32.785 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-08-18T21:53:32.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-18T21:53:32.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:53:32.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:53:32.785 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:53:32.813 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:32.810+0000 7f557f89a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:32.890 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:32.890+0000 7f557f89a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:32.892 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:32.890+0000 7f557f89a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:33.262 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-18T21:53:33.262 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:53:33.263 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:53:33.263 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:53:33.263 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:53:33.263 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:33.263 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:53:33.263 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:53:33.263 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:33.264 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:53:33.654 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:34.605 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:34.602+0000 7f557f89a7c0 -1 Falling back to public interface 2024-08-18T21:53:34.656 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:34.656 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:34.656 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:53:34.656 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:53:34.657 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:34.657 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:53:35.114 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:36.115 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:36.115 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:36.116 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:53:36.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:53:36.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:36.116 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:53:36.615 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:37.616 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:37.617 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:53:37.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:37.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:53:37.617 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:37.618 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:53:38.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:38.760 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:38.757+0000 7f557f89a7c0 -1 osd.1 30 log_to_monitors true 2024-08-18T21:53:39.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:39.046 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:39.046 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:53:39.047 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:53:39.047 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:39.047 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:53:39.654 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 40 up_thru 40 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/3517349701,v1:127.0.0.1:6811/3517349701] [v2:127.0.0.1:6812/3517349701,v1:127.0.0.1:6813/3517349701] exists,up 000c933c-86aa-4ddf-9c31-889e86e7877b 2024-08-18T21:53:39.654 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:53:39.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:53:39.655 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:53:39.655 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-08-18T21:53:39.657 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=121397 2024-08-18T21:53:39.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-08-18T21:53:39.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:53:39.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:53:39.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-18T21:53:39.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:53:39.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-08-18T21:53:39.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:53:39.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:53:39.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:53:39.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:53:39.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-08-18T21:53:39.659 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-08-18T21:53:39.659 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:53:39.659 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:53:39.659 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:53:39.659 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:53:39.659 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:53:39.659 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:53:39.659 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:53:39.659 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:53:39.659 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:53:39.659 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:53:39.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:53:39.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:53:39.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:53:39.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:53:39.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:53:39.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:53:39.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:53:39.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:53:39.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:53:39.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:53:39.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-08-18T21:53:39.661 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:53:39.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-18T21:53:39.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:53:39.662 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-08-18T21:53:39.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-18T21:53:39.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:53:39.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:53:39.662 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:53:39.689 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:39.685+0000 7fde04bfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:39.816 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:39.813+0000 7fde04bfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:39.817 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:39.813+0000 7fde04bfc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:40.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-18T21:53:40.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:53:40.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:53:40.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:53:40.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:53:40.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:40.239 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:53:40.240 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:53:40.240 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:40.240 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:53:40.686 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:41.345 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:41.341+0000 7fde04bfc7c0 -1 Falling back to public interface 2024-08-18T21:53:41.687 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:41.687 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:53:41.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:41.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:53:41.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:41.688 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:53:42.173 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:43.174 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:43.175 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:43.175 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:53:43.175 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:53:43.175 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:43.175 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:53:43.619 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:44.590 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:44.589+0000 7fde04bfc7c0 -1 osd.2 32 log_to_monitors true 2024-08-18T21:53:44.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:44.620 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:44.620 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:53:44.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:53:44.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:44.621 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:53:45.073 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:46.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:46.075 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:46.075 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:53:46.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:53:46.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:46.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:53:46.794 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 43 up_thru 43 down_at 33 last_clean_interval [15,32) [v2:127.0.0.1:6818/2462682237,v1:127.0.0.1:6819/2462682237] [v2:127.0.0.1:6820/2462682237,v1:127.0.0.1:6821/2462682237] exists,up dab4835d-5f3e-4530-8813-db3b51c8e6fa 2024-08-18T21:53:46.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:53:46.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:53:46.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:53:46.795 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-08-18T21:53:46.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=122370 2024-08-18T21:53:46.796 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-08-18T21:53:46.797 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:53:46.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:53:46.798 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:53:46.799 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:53:46.799 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:53:46.799 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:53:46.799 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:53:46.799 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:53:46.799 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:53:46.799 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:53:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:53:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:53:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:53:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:53:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:53:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:53:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:53:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:53:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:53:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:53:46.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-08-18T21:53:46.801 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.3 2024-08-18T21:53:46.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-18T21:53:46.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:53:46.801 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-08-18T21:53:46.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-18T21:53:46.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:53:46.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:53:46.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:53:46.832 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:46.829+0000 7f3ec84347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:46.835 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:46.833+0000 7f3ec84347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:46.837 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:46.833+0000 7f3ec84347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:53:47.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-18T21:53:47.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:53:47.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-18T21:53:47.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:53:47.490 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:53:47.491 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:53:47.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:47.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:53:47.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:47.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:53:48.023 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:48.409 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:48.405+0000 7f3ec84347c0 -1 Falling back to public interface 2024-08-18T21:53:49.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:49.024 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:49.024 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:53:49.025 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:53:49.025 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:49.025 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:53:49.459 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:50.461 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:50.461 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:50.461 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:53:50.462 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:53:50.462 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:50.462 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:53:50.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:51.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:51.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:51.914 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:53:51.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:53:51.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:51.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:53:52.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:53:52.864 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:53:52.861+0000 7f3ec84347c0 -1 osd.3 34 log_to_monitors true 2024-08-18T21:53:53.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:53:53.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:53:53.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:53:53.433 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:53:53.434 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:53:53.434 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:53:54.450 INFO:tasks.workunit.client.0.smithi099.stdout:osd.3 up in weight 1 up_from 48 up_thru 33 down_at 35 last_clean_interval [20,34) [v2:127.0.0.1:6826/1020567258,v1:127.0.0.1:6827/1020567258] [v2:127.0.0.1:6828/1020567258,v1:127.0.0.1:6829/1020567258] exists,up fc8c4dc2-5d7e-447d-b7fe-0621f00c25c1 2024-08-18T21:53:54.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:53:54.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:53:54.451 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:53:54.451 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-08-18T21:53:54.452 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=123322 2024-08-18T21:53:54.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-08-18T21:53:54.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:53:54.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:53:54.453 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:53:54.453 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-18T21:53:54.454 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:53:54.454 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:53:54.454 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:53:54.454 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:53:54.454 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:53:54.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-18T21:53:54.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:53:54.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:53:54.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:53:54.562 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:53:54.563 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:53:55.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:53:55.060 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:53:55.060 INFO:tasks.workunit.client.0.smithi099.stderr:2 2024-08-18T21:53:55.060 INFO:tasks.workunit.client.0.smithi099.stderr:3' 2024-08-18T21:53:55.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:53:55.060 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:53:55.060 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:53:55.316 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=158913789959 2024-08-18T21:53:55.316 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 158913789959 2024-08-18T21:53:55.316 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-158913789959' 2024-08-18T21:53:55.317 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:53:55.317 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:53:55.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=171798691846 2024-08-18T21:53:55.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 171798691846 2024-08-18T21:53:55.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-158913789959 1-171798691846' 2024-08-18T21:53:55.504 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:53:55.504 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:53:55.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=184683593733 2024-08-18T21:53:55.653 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 184683593733 2024-08-18T21:53:55.653 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-158913789959 1-171798691846 2-184683593733' 2024-08-18T21:53:55.653 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:53:55.653 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-18T21:53:55.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=206158430211 2024-08-18T21:53:55.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 206158430211 2024-08-18T21:53:55.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-158913789959 1-171798691846 2-184683593733 3-206158430211' 2024-08-18T21:53:55.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:53:55.805 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-158913789959 2024-08-18T21:53:55.805 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:53:55.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:53:55.807 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-158913789959 2024-08-18T21:53:55.807 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:53:55.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=158913789959 2024-08-18T21:53:55.808 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 158913789959 2024-08-18T21:53:55.809 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 158913789959' 2024-08-18T21:53:55.809 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:53:56.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 158913789957 -lt 158913789959 2024-08-18T21:53:56.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:53:57.224 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:53:57.225 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:53:57.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 158913789959 -lt 158913789959 2024-08-18T21:53:57.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:53:57.678 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:53:57.679 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-171798691846 2024-08-18T21:53:57.679 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:53:57.681 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-171798691846 2024-08-18T21:53:57.681 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:53:57.682 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=171798691846 2024-08-18T21:53:57.682 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 171798691846 2024-08-18T21:53:57.682 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 171798691846' 2024-08-18T21:53:57.683 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:53:58.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 171798691846 -lt 171798691846 2024-08-18T21:53:58.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:53:58.133 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-184683593733 2024-08-18T21:53:58.133 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:53:58.135 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:53:58.136 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-184683593733 2024-08-18T21:53:58.136 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:53:58.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=184683593733 2024-08-18T21:53:58.137 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 184683593733 2024-08-18T21:53:58.137 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 184683593733' 2024-08-18T21:53:58.137 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:53:58.597 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 184683593733 -lt 184683593733 2024-08-18T21:53:58.599 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:53:58.600 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-206158430211 2024-08-18T21:53:58.600 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:53:58.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=3 2024-08-18T21:53:58.601 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:53:58.601 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 3-206158430211 2024-08-18T21:53:58.602 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=206158430211 2024-08-18T21:53:58.603 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.3 seq 206158430211 2024-08-18T21:53:58.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.3 seq 206158430211' 2024-08-18T21:53:58.603 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-18T21:53:59.066 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 206158430212 -lt 206158430211 2024-08-18T21:53:59.067 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:53:59.067 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:53:59.067 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:53:59.648 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-08-18T21:53:59.648 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:53:59.649 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:53:59.649 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:53:59.649 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:53:59.649 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:53:59.649 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:53:59.650 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:54:00.095 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-08-18T21:54:00.096 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:54:00.096 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:54:00.096 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:54:00.745 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-08-18T21:54:00.746 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:54:00.746 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:54:00.746 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-08-18T21:54:00.781 INFO:tasks.workunit.client.0.smithi099.stdout:hints = 1 2024-08-18T21:54:00.781 INFO:tasks.workunit.client.0.smithi099.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-08-18T21:54:00.781 INFO:tasks.workunit.client.0.smithi099.stdout:Object prefix: benchmark_data_smithi099_124663 2024-08-18T21:54:00.781 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T21:54:00.781 INFO:tasks.workunit.client.0.smithi099.stdout: 0 0 0 0 0 0 - 0 2024-08-18T21:54:01.781 INFO:tasks.workunit.client.0.smithi099.stdout: 1 16 61 45 0.175759 0.175781 0.192531 0.290686 2024-08-18T21:54:02.781 INFO:tasks.workunit.client.0.smithi099.stdout: 2 16 111 95 0.185521 0.195312 0.359899 0.31117 2024-08-18T21:54:03.781 INFO:tasks.workunit.client.0.smithi099.stdout: 3 16 169 153 0.199191 0.226562 0.297664 0.299472 2024-08-18T21:54:04.781 INFO:tasks.workunit.client.0.smithi099.stdout: 4 16 221 205 0.200167 0.203125 0.385524 0.301535 2024-08-18T21:54:05.781 INFO:tasks.workunit.client.0.smithi099.stdout: 5 16 273 257 0.200752 0.203125 0.319881 0.300609 2024-08-18T21:54:06.781 INFO:tasks.workunit.client.0.smithi099.stdout: 6 16 328 312 0.203096 0.214844 0.298789 0.299521 2024-08-18T21:54:07.781 INFO:tasks.workunit.client.0.smithi099.stdout: 7 16 375 359 0.200307 0.183594 0.219479 0.304598 2024-08-18T21:54:08.781 INFO:tasks.workunit.client.0.smithi099.stdout: 8 16 433 417 0.203584 0.226562 0.235971 0.300141 2024-08-18T21:54:09.782 INFO:tasks.workunit.client.0.smithi099.stdout: 9 16 490 474 0.2057 0.222656 0.294316 0.300398 2024-08-18T21:54:10.782 INFO:tasks.workunit.client.0.smithi099.stdout: 10 16 540 524 0.204658 0.195312 0.402339 0.30037 2024-08-18T21:54:11.782 INFO:tasks.workunit.client.0.smithi099.stdout: 11 16 591 575 0.204162 0.199219 0.336301 0.30269 2024-08-18T21:54:12.782 INFO:tasks.workunit.client.0.smithi099.stdout: 12 16 635 619 0.201469 0.171875 0.290927 0.302939 2024-08-18T21:54:13.782 INFO:tasks.workunit.client.0.smithi099.stdout: 13 16 690 674 0.202496 0.214844 0.297331 0.304645 2024-08-18T21:54:14.782 INFO:tasks.workunit.client.0.smithi099.stdout: 14 16 747 731 0.203934 0.222656 0.240595 0.304253 2024-08-18T21:54:15.783 INFO:tasks.workunit.client.0.smithi099.stdout: 15 16 800 784 0.204139 0.207031 0.234723 0.302936 2024-08-18T21:54:16.783 INFO:tasks.workunit.client.0.smithi099.stdout: 16 16 843 827 0.201878 0.167969 0.291393 0.304337 2024-08-18T21:54:17.783 INFO:tasks.workunit.client.0.smithi099.stdout: 17 16 897 881 0.202409 0.210938 0.259721 0.305879 2024-08-18T21:54:18.783 INFO:tasks.workunit.client.0.smithi099.stdout: 18 16 948 932 0.20223 0.199219 0.36055 0.3071 2024-08-18T21:54:19.783 INFO:tasks.workunit.client.0.smithi099.stdout: 19 16 1004 988 0.203098 0.21875 0.21233 0.305804 2024-08-18T21:54:20.783 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:54:20.786562+0000 min lat: 0.142278 max lat: 0.41836 avg lat: 0.300172 2024-08-18T21:54:20.783 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T21:54:20.783 INFO:tasks.workunit.client.0.smithi099.stdout: 20 16 1076 1060 0.207004 0.28125 0.183759 0.300172 2024-08-18T21:54:21.783 INFO:tasks.workunit.client.0.smithi099.stdout: 21 16 1145 1129 0.20998 0.269531 0.216924 0.296322 2024-08-18T21:54:22.783 INFO:tasks.workunit.client.0.smithi099.stdout: 22 16 1206 1190 0.211265 0.238281 0.300907 0.294317 2024-08-18T21:54:23.784 INFO:tasks.workunit.client.0.smithi099.stdout: 23 16 1267 1251 0.212438 0.238281 0.281432 0.292517 2024-08-18T21:54:24.784 INFO:tasks.workunit.client.0.smithi099.stdout: 24 16 1330 1314 0.21384 0.246094 0.277508 0.290564 2024-08-18T21:54:25.784 INFO:tasks.workunit.client.0.smithi099.stdout: 25 16 1394 1378 0.215285 0.25 0.243763 0.289362 2024-08-18T21:54:26.784 INFO:tasks.workunit.client.0.smithi099.stdout: 26 16 1452 1436 0.215717 0.226562 0.237053 0.288123 2024-08-18T21:54:27.784 INFO:tasks.workunit.client.0.smithi099.stdout: 27 16 1509 1493 0.215973 0.222656 0.274944 0.287244 2024-08-18T21:54:28.784 INFO:tasks.workunit.client.0.smithi099.stdout: 28 16 1572 1556 0.217047 0.246094 0.27786 0.28678 2024-08-18T21:54:29.784 INFO:tasks.workunit.client.0.smithi099.stdout: 29 16 1634 1618 0.217913 0.242188 0.266955 0.285759 2024-08-18T21:54:30.784 INFO:tasks.workunit.client.0.smithi099.stdout: 30 16 1688 1672 0.21768 0.210938 0.275323 0.285302 2024-08-18T21:54:31.785 INFO:tasks.workunit.client.0.smithi099.stdout:Total time run: 30.1784 2024-08-18T21:54:31.785 INFO:tasks.workunit.client.0.smithi099.stdout:Total writes made: 1688 2024-08-18T21:54:31.785 INFO:tasks.workunit.client.0.smithi099.stdout:Write size: 4096 2024-08-18T21:54:31.785 INFO:tasks.workunit.client.0.smithi099.stdout:Object size: 4096 2024-08-18T21:54:31.785 INFO:tasks.workunit.client.0.smithi099.stdout:Bandwidth (MB/sec): 0.218492 2024-08-18T21:54:31.785 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Bandwidth: 0.0268864 2024-08-18T21:54:31.785 INFO:tasks.workunit.client.0.smithi099.stdout:Max bandwidth (MB/sec): 0.28125 2024-08-18T21:54:31.785 INFO:tasks.workunit.client.0.smithi099.stdout:Min bandwidth (MB/sec): 0.167969 2024-08-18T21:54:31.785 INFO:tasks.workunit.client.0.smithi099.stdout:Average IOPS: 55 2024-08-18T21:54:31.786 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev IOPS: 6.88293 2024-08-18T21:54:31.786 INFO:tasks.workunit.client.0.smithi099.stdout:Max IOPS: 72 2024-08-18T21:54:31.786 INFO:tasks.workunit.client.0.smithi099.stdout:Min IOPS: 43 2024-08-18T21:54:31.786 INFO:tasks.workunit.client.0.smithi099.stdout:Average Latency(s): 0.285196 2024-08-18T21:54:31.786 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Latency(s): 0.0543841 2024-08-18T21:54:31.786 INFO:tasks.workunit.client.0.smithi099.stdout:Max latency(s): 0.41836 2024-08-18T21:54:31.786 INFO:tasks.workunit.client.0.smithi099.stdout:Min latency(s): 0.133604 2024-08-18T21:54:32.007 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 120408 2024-08-18T21:54:32.007 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-08-18T21:54:33.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 120408 2024-08-18T21:54:33.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (120408) - No such process 2024-08-18T21:54:33.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-08-18T21:54:33.473 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already down. 2024-08-18T21:54:33.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 121397 2024-08-18T21:54:33.489 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-08-18T21:54:34.491 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 121397 2024-08-18T21:54:34.492 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (121397) - No such process 2024-08-18T21:54:34.492 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-08-18T21:54:34.939 INFO:tasks.workunit.client.0.smithi099.stderr:osd.1 is already down. 2024-08-18T21:54:34.957 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 122370 2024-08-18T21:54:34.958 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-08-18T21:54:35.957 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 122370 2024-08-18T21:54:35.957 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (122370) - No such process 2024-08-18T21:54:35.957 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-08-18T21:54:36.414 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already down. 2024-08-18T21:54:36.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 123322 2024-08-18T21:54:36.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-08-18T21:54:37.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 123322 2024-08-18T21:54:37.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (123322) - No such process 2024-08-18T21:54:37.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-08-18T21:54:38.000 INFO:tasks.workunit.client.0.smithi099.stderr:osd.3 is already down. 2024-08-18T21:54:38.017 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-18T21:54:39.906 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:54:39.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-08-18T21:54:40.229 INFO:tasks.workunit.client.0.smithi099.stderr:512+0 records in 2024-08-18T21:54:40.229 INFO:tasks.workunit.client.0.smithi099.stderr:512+0 records out 2024-08-18T21:54:40.229 INFO:tasks.workunit.client.0.smithi099.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.302604 s, 1.8 GB/s 2024-08-18T21:54:40.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-08-18T21:54:40.253 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:54:45.886 INFO:tasks.workunit.client.0.smithi099.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-08-18T21:54:45.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-18T21:54:48.262 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:54:48.280 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-18T21:54:50.151 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:54:50.165 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-08-18T21:54:50.731 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records in 2024-08-18T21:54:50.731 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records out 2024-08-18T21:54:50.731 INFO:tasks.workunit.client.0.smithi099.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.564675 s, 1.9 GB/s 2024-08-18T21:54:50.731 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-08-18T21:54:50.755 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:54:59.631 INFO:tasks.workunit.client.0.smithi099.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-08-18T21:54:59.636 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-08-18T21:54:59.663 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:55:00.802 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:00.799+0000 7fc5a4600ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:01.591 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:01.591+0000 7fc5a4600ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:02.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-18T21:55:03.176 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:03.175+0000 7f67cc195ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:03.971 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:03.971+0000 7f67cc195ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:04.946 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:55:04.965 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-18T21:55:06.330 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:55:06.340 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-08-18T21:55:06.366 INFO:tasks.workunit.client.0.smithi099.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-08-18T21:55:06.366 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:55:07.722 INFO:tasks.workunit.client.0.smithi099.stdout:DB device added 2024-08-18T21:55:07.728 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-08-18T21:55:07.754 INFO:tasks.workunit.client.0.smithi099.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-08-18T21:55:07.754 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:55:08.652 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:08.651+0000 7f5ced977ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:09.170 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:09.167+0000 7f5ced977ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:09.244 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:09.243+0000 7f5ced977ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:09.894 INFO:tasks.workunit.client.0.smithi099.stdout:WAL device added 2024-08-18T21:55:09.900 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-08-18T21:55:09.927 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:55:11.060 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:11.059+0000 7fa49a68aac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:11.846 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:11.847+0000 7fa49a68aac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:13.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-18T21:55:13.689 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:13.687+0000 7f2790e70ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:14.487 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:14.487+0000 7f2790e70ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:15.474 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:55:15.493 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-18T21:55:17.371 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:55:17.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-08-18T21:55:17.414 INFO:tasks.workunit.client.0.smithi099.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-08-18T21:55:17.414 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:55:19.547 INFO:tasks.workunit.client.0.smithi099.stdout:WAL device added 2024-08-18T21:55:19.552 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-08-18T21:55:19.579 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:55:22.775 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-08-18T21:55:22.784 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-18T21:55:24.655 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:55:24.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-08-18T21:55:24.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:55:24.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:55:24.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-18T21:55:24.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:55:24.672 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-18T21:55:24.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:55:24.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:55:24.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:55:24.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:55:24.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-18T21:55:24.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-18T21:55:24.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:55:24.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:55:24.673 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:55:24.674 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:55:24.674 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:55:24.674 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:55:24.674 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:55:24.674 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:55:24.674 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:55:24.675 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:55:24.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:55:24.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:55:24.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:55:24.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:55:24.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:55:24.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:55:24.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:55:24.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:55:24.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:55:24.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:55:24.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-18T21:55:24.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-18T21:55:24.677 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:55:24.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:55:24.678 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-08-18T21:55:24.679 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-18T21:55:24.679 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:55:24.679 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:55:24.680 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:55:24.715 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:24.711+0000 7fefc19797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:24.715 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:24.711+0000 7fefc19797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:24.715 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:24.715+0000 7fefc19797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:25.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-18T21:55:25.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:55:25.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:55:25.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:55:25.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:55:25.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:25.098 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:55:25.098 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:55:25.098 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:25.098 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:55:25.530 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:26.532 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:26.532 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:26.532 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:55:26.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:55:26.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:26.533 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:55:26.700 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:26.699+0000 7fefc19797c0 -1 Falling back to public interface 2024-08-18T21:55:26.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:27.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:27.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:27.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:55:27.968 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:55:27.968 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:27.968 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:55:28.414 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:29.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:29.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:29.416 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:55:29.416 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:55:29.417 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:29.417 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:55:29.876 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:30.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:30.878 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:30.879 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:55:30.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:55:30.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:30.879 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:55:31.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:31.992 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:31.990+0000 7fefc19797c0 -1 osd.0 50 log_to_monitors true 2024-08-18T21:55:32.306 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:32.307 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:32.307 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:55:32.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:55:32.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:32.308 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:55:32.798 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 60 up_thru 60 down_at 51 last_clean_interval [37,50) [v2:127.0.0.1:6802/4019642516,v1:127.0.0.1:6803/4019642516] [v2:127.0.0.1:6804/4019642516,v1:127.0.0.1:6805/4019642516] exists,up 0070fe81-9b0d-489d-b9a5-ad08cec4b62e 2024-08-18T21:55:32.799 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:55:32.799 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:55:32.799 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:55:32.800 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-18T21:55:32.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=130818 2024-08-18T21:55:32.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-08-18T21:55:32.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:55:32.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:55:32.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-18T21:55:32.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:55:32.804 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:55:32.805 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:55:32.806 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:55:32.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:55:32.807 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-08-18T21:55:32.807 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:55:32.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-18T21:55:32.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:55:32.808 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-08-18T21:55:32.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-18T21:55:32.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:55:32.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:55:32.808 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:55:32.831 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:32.830+0000 7f075da637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:32.847 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:32.846+0000 7f075da637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:32.848 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:32.846+0000 7f075da637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:33.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-18T21:55:33.281 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:55:33.282 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:55:33.282 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:55:33.282 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:55:33.282 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:55:33.282 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:33.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:55:33.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:33.283 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:55:33.699 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:34.701 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:34.701 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:34.701 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:55:34.701 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:55:34.702 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:34.702 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:55:34.879 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:34.878+0000 7f075da637c0 -1 Falling back to public interface 2024-08-18T21:55:35.120 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:36.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:36.122 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:36.123 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:55:36.123 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:55:36.124 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:36.124 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:55:36.546 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:36.814 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:36.810+0000 7f075da637c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:37.370 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:37.370+0000 7f075da637c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:37.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:37.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:37.547 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:55:37.548 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:55:37.548 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:55:37.548 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:38.006 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:38.328 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:38.326+0000 7f075da637c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:39.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:39.008 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:39.008 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:55:39.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:55:39.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:39.009 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:55:39.136 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:39.134+0000 7f075da637c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:39.446 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:39.615 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:39.614+0000 7f075da637c0 -1 osd.1 53 log_to_monitors true 2024-08-18T21:55:40.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:40.448 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:40.448 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:55:40.449 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:55:40.449 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:40.449 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:55:40.922 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 63 up_thru 63 down_at 54 last_clean_interval [40,53) [v2:127.0.0.1:6810/1185112502,v1:127.0.0.1:6811/1185112502] [v2:127.0.0.1:6812/1185112502,v1:127.0.0.1:6813/1185112502] exists,up 000c933c-86aa-4ddf-9c31-889e86e7877b 2024-08-18T21:55:40.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:55:40.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:55:40.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:55:40.923 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-08-18T21:55:40.924 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=131821 2024-08-18T21:55:40.924 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-08-18T21:55:40.924 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:55:40.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:55:40.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-18T21:55:40.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:55:40.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-08-18T21:55:40.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:55:40.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:55:40.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:55:40.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:55:40.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-08-18T21:55:40.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-08-18T21:55:40.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:55:40.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:55:40.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:55:40.928 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:55:40.928 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:55:40.928 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:55:40.928 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:55:40.928 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:55:40.928 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:55:40.928 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:55:40.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:55:40.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:55:40.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:55:40.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:55:40.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:55:40.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:55:40.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:55:40.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:55:40.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:55:40.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:55:40.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-08-18T21:55:40.930 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:55:40.930 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-18T21:55:40.931 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:55:40.931 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-08-18T21:55:40.931 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-18T21:55:40.931 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:55:40.931 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:55:40.932 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:55:40.953 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:40.950+0000 7fb5b9c2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:41.041 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:41.038+0000 7fb5b9c2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:41.043 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:41.042+0000 7fb5b9c2c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:41.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-18T21:55:41.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:55:41.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:55:41.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:55:41.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:55:41.376 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:41.376 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:55:41.377 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:55:41.377 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:41.377 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:55:41.812 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:42.793 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:42.790+0000 7fb5b9c2c7c0 -1 Falling back to public interface 2024-08-18T21:55:42.813 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:42.813 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:42.813 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:55:42.813 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:55:42.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:42.814 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:55:43.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:44.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:44.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:44.228 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:55:44.229 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:55:44.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:44.229 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:55:44.646 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:44.934 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:44.930+0000 7fb5b9c2c7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:45.648 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:45.648 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:45.648 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:55:45.649 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:55:45.649 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:45.649 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:55:45.735 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:45.734+0000 7fb5b9c2c7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:46.106 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:46.744 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:46.742+0000 7fb5b9c2c7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:47.108 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:47.108 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:47.108 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:55:47.109 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:55:47.109 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:47.109 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:55:47.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:47.558 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:47.558+0000 7fb5b9c2c7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:55:48.543 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:48.543 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:48.543 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:55:48.543 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:55:48.544 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:48.544 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:55:48.592 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:48.590+0000 7fb5b9c2c7c0 -1 osd.2 55 log_to_monitors true 2024-08-18T21:55:48.978 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:49.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:49.979 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:49.980 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:55:49.980 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:55:49.980 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:49.980 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:55:50.562 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 67 up_thru 67 down_at 56 last_clean_interval [43,55) [v2:127.0.0.1:6818/1372787479,v1:127.0.0.1:6819/1372787479] [v2:127.0.0.1:6820/1372787479,v1:127.0.0.1:6821/1372787479] exists,up dab4835d-5f3e-4530-8813-db3b51c8e6fa 2024-08-18T21:55:50.563 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:55:50.563 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:55:50.563 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:55:50.563 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-08-18T21:55:50.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=132838 2024-08-18T21:55:50.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-08-18T21:55:50.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:55:50.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:55:50.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-18T21:55:50.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:55:50.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-08-18T21:55:50.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:55:50.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:55:50.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:55:50.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:55:50.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-08-18T21:55:50.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-08-18T21:55:50.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:55:50.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:55:50.571 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:55:50.571 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:55:50.571 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:55:50.571 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:55:50.571 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:55:50.571 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:55:50.571 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:55:50.571 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-08-18T21:55:50.572 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-18T21:55:50.572 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.3 2024-08-18T21:55:50.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:55:50.573 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-08-18T21:55:50.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-18T21:55:50.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:55:50.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:55:50.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:55:50.597 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:50.594+0000 7effb78cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:50.599 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:50.598+0000 7effb78cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:50.601 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:50.598+0000 7effb78cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:55:51.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-18T21:55:51.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:55:51.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-18T21:55:51.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:55:51.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:55:51.203 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:51.204 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:55:51.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:55:51.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:51.204 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:55:51.675 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:52.141 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:52.138+0000 7effb78cc7c0 -1 Falling back to public interface 2024-08-18T21:55:52.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:52.676 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:52.677 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:55:52.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:55:52.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:52.677 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:55:53.091 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:54.092 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:54.092 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:54.093 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:55:54.093 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:55:54.093 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:54.093 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:55:54.511 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:55.512 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:55.513 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:55.513 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:55:55.513 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:55:55.514 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:55.514 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:55:55.964 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:56.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:56.966 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:56.966 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:55:56.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:55:56.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:56.967 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:55:57.011 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:55:57.010+0000 7effb78cc7c0 -1 osd.3 57 log_to_monitors true 2024-08-18T21:55:57.429 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:55:58.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:55:58.431 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:55:58.431 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:55:58.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:55:58.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:55:58.432 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:55:59.154 INFO:tasks.workunit.client.0.smithi099.stdout:osd.3 up in weight 1 up_from 72 up_thru 72 down_at 58 last_clean_interval [48,57) [v2:127.0.0.1:6826/1451880965,v1:127.0.0.1:6827/1451880965] [v2:127.0.0.1:6828/1451880965,v1:127.0.0.1:6829/1451880965] exists,up fc8c4dc2-5d7e-447d-b7fe-0621f00c25c1 2024-08-18T21:55:59.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:55:59.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:55:59.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:55:59.155 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-08-18T21:55:59.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=133852 2024-08-18T21:55:59.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-08-18T21:55:59.194 INFO:tasks.workunit.client.0.smithi099.stdout:hints = 1 2024-08-18T21:55:59.194 INFO:tasks.workunit.client.0.smithi099.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-08-18T21:55:59.194 INFO:tasks.workunit.client.0.smithi099.stdout:Object prefix: benchmark_data_smithi099_134835 2024-08-18T21:55:59.195 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T21:55:59.195 INFO:tasks.workunit.client.0.smithi099.stdout: 0 0 0 0 0 0 - 0 2024-08-18T21:56:00.195 INFO:tasks.workunit.client.0.smithi099.stdout: 1 16 48 32 0.124987 0.125 0.406619 0.367593 2024-08-18T21:56:01.195 INFO:tasks.workunit.client.0.smithi099.stdout: 2 16 95 79 0.154276 0.183594 0.309564 0.376442 2024-08-18T21:56:02.195 INFO:tasks.workunit.client.0.smithi099.stdout: 3 16 143 127 0.165344 0.1875 0.408626 0.36219 2024-08-18T21:56:03.195 INFO:tasks.workunit.client.0.smithi099.stdout: 4 16 188 172 0.167947 0.175781 0.208907 0.357384 2024-08-18T21:56:04.195 INFO:tasks.workunit.client.0.smithi099.stdout: 5 16 231 215 0.167947 0.167969 0.546636 0.35253 2024-08-18T21:56:05.195 INFO:tasks.workunit.client.0.smithi099.stdout: 6 16 275 259 0.168597 0.171875 0.35294 0.358231 2024-08-18T21:56:06.196 INFO:tasks.workunit.client.0.smithi099.stdout: 7 16 326 310 0.172967 0.199219 0.245207 0.350021 2024-08-18T21:56:07.195 INFO:tasks.workunit.client.0.smithi099.stdout: 8 16 373 357 0.174293 0.183594 0.275559 0.349923 2024-08-18T21:56:08.196 INFO:tasks.workunit.client.0.smithi099.stdout: 9 16 424 408 0.177059 0.199219 0.351502 0.345825 2024-08-18T21:56:09.196 INFO:tasks.workunit.client.0.smithi099.stdout: 10 16 467 451 0.176148 0.167969 0.375866 0.34858 2024-08-18T21:56:10.196 INFO:tasks.workunit.client.0.smithi099.stdout: 11 16 512 496 0.176113 0.175781 0.227143 0.346443 2024-08-18T21:56:11.196 INFO:tasks.workunit.client.0.smithi099.stdout: 12 16 564 548 0.178362 0.203125 0.206193 0.346274 2024-08-18T21:56:12.196 INFO:tasks.workunit.client.0.smithi099.stdout: 13 16 606 590 0.17726 0.164062 0.372127 0.347385 2024-08-18T21:56:13.196 INFO:tasks.workunit.client.0.smithi099.stdout: 14 16 650 634 0.176874 0.171875 0.210235 0.348406 2024-08-18T21:56:14.196 INFO:tasks.workunit.client.0.smithi099.stdout: 15 16 696 680 0.17706 0.179688 0.209545 0.347556 2024-08-18T21:56:15.197 INFO:tasks.workunit.client.0.smithi099.stdout: 16 16 733 717 0.175025 0.144531 0.209624 0.352011 2024-08-18T21:56:16.197 INFO:tasks.workunit.client.0.smithi099.stdout: 17 16 787 771 0.177136 0.210938 0.334093 0.348894 2024-08-18T21:56:17.197 INFO:tasks.workunit.client.0.smithi099.stdout: 18 16 830 814 0.176625 0.167969 0.422355 0.350008 2024-08-18T21:56:18.197 INFO:tasks.workunit.client.0.smithi099.stdout: 19 16 880 864 0.177607 0.195312 0.333906 0.349308 2024-08-18T21:56:19.197 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:56:19.200394+0000 min lat: 0.191627 max lat: 0.988726 avg lat: 0.349658 2024-08-18T21:56:19.198 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T21:56:19.198 INFO:tasks.workunit.client.0.smithi099.stdout: 20 16 921 905 0.176733 0.160156 0.218927 0.349658 2024-08-18T21:56:20.197 INFO:tasks.workunit.client.0.smithi099.stdout: 21 16 968 952 0.177058 0.183594 0.409138 0.349286 2024-08-18T21:56:21.198 INFO:tasks.workunit.client.0.smithi099.stdout: 22 16 1016 1000 0.177532 0.1875 0.365861 0.349563 2024-08-18T21:56:22.198 INFO:tasks.workunit.client.0.smithi099.stdout: 23 16 1068 1052 0.178643 0.203125 0.218014 0.348266 2024-08-18T21:56:23.198 INFO:tasks.workunit.client.0.smithi099.stdout: 24 16 1111 1095 0.178197 0.167969 0.390169 0.348922 2024-08-18T21:56:24.198 INFO:tasks.workunit.client.0.smithi099.stdout: 25 16 1150 1134 0.177162 0.152344 0.226653 0.349034 2024-08-18T21:56:25.198 INFO:tasks.workunit.client.0.smithi099.stdout: 26 16 1196 1180 0.177258 0.179688 0.208629 0.351252 2024-08-18T21:56:26.198 INFO:tasks.workunit.client.0.smithi099.stdout: 27 16 1238 1222 0.176768 0.164062 0.231432 0.350308 2024-08-18T21:56:27.199 INFO:tasks.workunit.client.0.smithi099.stdout: 28 16 1285 1269 0.177011 0.183594 0.35763 0.351346 2024-08-18T21:56:28.199 INFO:tasks.workunit.client.0.smithi099.stdout: 29 16 1333 1317 0.177372 0.1875 0.234609 0.350541 2024-08-18T21:56:29.199 INFO:tasks.workunit.client.0.smithi099.stdout: 30 16 1374 1358 0.176797 0.160156 0.215645 0.351098 2024-08-18T21:56:30.199 INFO:tasks.workunit.client.0.smithi099.stdout:Total time run: 30.1942 2024-08-18T21:56:30.199 INFO:tasks.workunit.client.0.smithi099.stdout:Total writes made: 1374 2024-08-18T21:56:30.199 INFO:tasks.workunit.client.0.smithi099.stdout:Write size: 4096 2024-08-18T21:56:30.199 INFO:tasks.workunit.client.0.smithi099.stdout:Object size: 4096 2024-08-18T21:56:30.199 INFO:tasks.workunit.client.0.smithi099.stdout:Bandwidth (MB/sec): 0.177755 2024-08-18T21:56:30.199 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Bandwidth: 0.0185767 2024-08-18T21:56:30.199 INFO:tasks.workunit.client.0.smithi099.stdout:Max bandwidth (MB/sec): 0.210938 2024-08-18T21:56:30.200 INFO:tasks.workunit.client.0.smithi099.stdout:Min bandwidth (MB/sec): 0.125 2024-08-18T21:56:30.200 INFO:tasks.workunit.client.0.smithi099.stdout:Average IOPS: 45 2024-08-18T21:56:30.200 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev IOPS: 4.75564 2024-08-18T21:56:30.200 INFO:tasks.workunit.client.0.smithi099.stdout:Max IOPS: 54 2024-08-18T21:56:30.200 INFO:tasks.workunit.client.0.smithi099.stdout:Min IOPS: 32 2024-08-18T21:56:30.200 INFO:tasks.workunit.client.0.smithi099.stdout:Average Latency(s): 0.350788 2024-08-18T21:56:30.200 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Latency(s): 0.0695728 2024-08-18T21:56:30.200 INFO:tasks.workunit.client.0.smithi099.stdout:Max latency(s): 0.988726 2024-08-18T21:56:30.200 INFO:tasks.workunit.client.0.smithi099.stdout:Min latency(s): 0.191627 2024-08-18T21:56:30.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 130818 2024-08-18T21:56:30.344 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-08-18T21:56:31.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 130818 2024-08-18T21:56:31.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (130818) - No such process 2024-08-18T21:56:31.346 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-08-18T21:56:31.936 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already down. 2024-08-18T21:56:31.953 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 131821 2024-08-18T21:56:31.953 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-08-18T21:56:32.956 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 131821 2024-08-18T21:56:32.956 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (131821) - No such process 2024-08-18T21:56:32.956 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-08-18T21:56:33.477 INFO:tasks.workunit.client.0.smithi099.stderr:osd.1 is already down. 2024-08-18T21:56:33.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 132838 2024-08-18T21:56:33.495 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-08-18T21:56:34.496 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 132838 2024-08-18T21:56:34.496 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (132838) - No such process 2024-08-18T21:56:34.496 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-08-18T21:56:35.344 INFO:tasks.workunit.client.0.smithi099.stderr:osd.2 is already down. 2024-08-18T21:56:35.360 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 133852 2024-08-18T21:56:35.360 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-08-18T21:56:36.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 133852 2024-08-18T21:56:36.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (133852) - No such process 2024-08-18T21:56:36.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-08-18T21:56:36.955 INFO:tasks.workunit.client.0.smithi099.stderr:osd.3 is already down. 2024-08-18T21:56:36.971 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-18T21:56:39.334 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:56:39.356 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-08-18T21:56:39.920 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records in 2024-08-18T21:56:39.920 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records out 2024-08-18T21:56:39.920 INFO:tasks.workunit.client.0.smithi099.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.562875 s, 1.9 GB/s 2024-08-18T21:56:39.920 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-08-18T21:56:39.944 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:56:50.379 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-08-18T21:56:50.379 INFO:tasks.workunit.client.0.smithi099.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-08-18T21:56:50.384 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-18T21:56:52.755 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:56:52.774 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-08-18T21:56:53.079 INFO:tasks.workunit.client.0.smithi099.stderr:512+0 records in 2024-08-18T21:56:53.079 INFO:tasks.workunit.client.0.smithi099.stderr:512+0 records out 2024-08-18T21:56:53.079 INFO:tasks.workunit.client.0.smithi099.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.3036 s, 1.8 GB/s 2024-08-18T21:56:53.079 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-08-18T21:56:53.104 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:57:00.018 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-08-18T21:57:00.019 INFO:tasks.workunit.client.0.smithi099.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-08-18T21:57:00.027 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-08-18T21:57:00.094 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-08-18T21:57:02.475 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:57:02.496 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-18T21:57:03.099 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:03.100+0000 7f0161a39ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:03.891 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:03.892+0000 7f0161a39ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:04.858 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:57:04.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-08-18T21:57:05.440 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records in 2024-08-18T21:57:05.440 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records out 2024-08-18T21:57:05.440 INFO:tasks.workunit.client.0.smithi099.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.55588 s, 1.9 GB/s 2024-08-18T21:57:05.440 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-08-18T21:57:05.464 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:57:06.624 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:06.624+0000 7fb827de4ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:07.407 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:07.408+0000 7fb827de4ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:14.233 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:14.232+0000 7fb827de4ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:15.902 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-08-18T21:57:15.903 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-08-18T21:57:15.903 INFO:tasks.workunit.client.0.smithi099.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-08-18T21:57:15.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-08-18T21:57:16.028 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-18T21:57:16.624 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:16.624+0000 7f4ba8096ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:17.154 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:17.152+0000 7f4ba8096ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:17.899 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:57:17.915 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-08-18T21:57:17.941 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:57:18.787 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:18.788+0000 7f5f650d0ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:19.322 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:19.320+0000 7f5f650d0ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:20.063 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-08-18T21:57:20.072 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-08-18T21:57:20.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-08-18T21:57:21.574 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:57:21.589 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-08-18T21:57:23.682 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:57:23.707 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-08-18T21:57:24.304 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records in 2024-08-18T21:57:24.305 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records out 2024-08-18T21:57:24.305 INFO:tasks.workunit.client.0.smithi099.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.595602 s, 1.8 GB/s 2024-08-18T21:57:24.305 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-08-18T21:57:24.328 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:57:25.182 INFO:tasks.workunit.client.0.smithi099.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-08-18T21:57:25.185 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-08-18T21:57:25.185 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-18T21:57:25.783 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:25.784+0000 7f77408eaac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:26.567 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:26.568+0000 7f77408eaac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:27.558 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:57:27.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-08-18T21:57:35.573 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records in 2024-08-18T21:57:35.573 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records out 2024-08-18T21:57:35.573 INFO:tasks.workunit.client.0.smithi099.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.85508 s, 137 MB/s 2024-08-18T21:57:35.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-08-18T21:57:35.600 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:57:36.784 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:36.783+0000 7fcb7e4beac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:37.555 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:37.555+0000 7fcb7e4beac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:37.621 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:37.619+0000 7fcb7e4beac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:39.026 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-08-18T21:57:39.027 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-08-18T21:57:39.027 INFO:tasks.workunit.client.0.smithi099.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-08-18T21:57:39.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-08-18T21:57:39.640 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:39.639+0000 7fab46c3cac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:40.174 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:40.171+0000 7fab46c3cac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:40.902 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:57:40.919 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-08-18T21:57:41.509 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records in 2024-08-18T21:57:41.509 INFO:tasks.workunit.client.0.smithi099.stderr:1024+0 records out 2024-08-18T21:57:41.509 INFO:tasks.workunit.client.0.smithi099.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.588021 s, 1.8 GB/s 2024-08-18T21:57:41.509 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-08-18T21:57:41.532 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T21:57:51.170 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-08-18T21:57:51.171 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-08-18T21:57:51.171 INFO:tasks.workunit.client.0.smithi099.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-08-18T21:57:51.180 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-08-18T21:57:51.776 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:51.775+0000 7f25b9213ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:52.348 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:52.347+0000 7f25b9213ac0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:57:53.050 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T21:57:53.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-08-18T21:57:53.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:57:53.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:57:53.064 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-18T21:57:53.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:57:53.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-18T21:57:53.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:57:53.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:57:53.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:57:53.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:57:53.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-18T21:57:53.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-18T21:57:53.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:57:53.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:57:53.065 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:57:53.066 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:57:53.066 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:57:53.066 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:57:53.067 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:57:53.067 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:57:53.067 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:57:53.067 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:57:53.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:57:53.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:57:53.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:57:53.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:57:53.068 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:57:53.068 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:57:53.068 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:57:53.068 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:57:53.068 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:57:53.068 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:57:53.068 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-18T21:57:53.068 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:57:53.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-18T21:57:53.069 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:57:53.069 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-08-18T21:57:53.070 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-18T21:57:53.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:57:53.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:57:53.071 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:57:53.100 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:53.099+0000 7feef393f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:57:53.103 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:53.103+0000 7feef393f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:57:53.104 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:53.103+0000 7feef393f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:57:53.545 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-18T21:57:53.545 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:57:53.545 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:57:53.546 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:57:53.546 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:57:53.546 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:57:53.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:57:53.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:57:53.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:57:53.547 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:57:53.940 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:57:54.637 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:54.635+0000 7feef393f7c0 -1 Falling back to public interface 2024-08-18T21:57:54.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:57:54.942 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:57:54.943 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:57:54.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:57:54.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:57:54.943 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:57:55.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:57:56.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:57:56.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:57:56.371 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:57:56.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:57:56.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:57:56.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:57:56.792 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:57:57.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:57:57.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:57:57.794 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:57:57.795 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:57:57.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:57:57.795 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:57:58.223 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:57:59.225 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:57:59.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:57:59.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:57:59.226 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:57:59.226 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:57:59.227 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:57:59.554 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:57:59.551+0000 7feef393f7c0 -1 osd.0 75 log_to_monitors true 2024-08-18T21:57:59.718 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:00.720 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:00.720 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:00.720 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:58:00.721 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:58:00.721 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:00.721 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:58:01.146 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 86 up_thru 86 down_at 76 last_clean_interval [60,75) [v2:127.0.0.1:6802/2495610532,v1:127.0.0.1:6803/2495610532] [v2:127.0.0.1:6804/2495610532,v1:127.0.0.1:6805/2495610532] exists,up 0070fe81-9b0d-489d-b9a5-ad08cec4b62e 2024-08-18T21:58:01.146 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:58:01.146 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:58:01.146 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:58:01.147 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-18T21:58:01.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=141009 2024-08-18T21:58:01.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-08-18T21:58:01.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:58:01.150 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:58:01.151 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:58:01.153 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:58:01.153 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:58:01.153 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:58:01.153 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:58:01.153 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:58:01.154 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-08-18T21:58:01.155 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.1 2024-08-18T21:58:01.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-08-18T21:58:01.155 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:58:01.156 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-08-18T21:58:01.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-08-18T21:58:01.156 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:58:01.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:58:01.157 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:58:01.184 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:01.183+0000 7fddb6c497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:58:01.188 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:01.187+0000 7fddb6c497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:58:01.189 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:01.187+0000 7fddb6c497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:58:01.605 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-08-18T21:58:01.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:58:01.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-08-18T21:58:01.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:58:01.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:58:01.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:01.606 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:58:01.607 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:58:01.607 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:58:01.607 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:02.074 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:02.417 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:02.415+0000 7fddb6c497c0 -1 Falling back to public interface 2024-08-18T21:58:03.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:03.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:03.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:58:03.077 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:58:03.077 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:03.077 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:58:03.497 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:04.497 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:04.497 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:04.497 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:58:04.498 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:58:04.498 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:04.498 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:58:04.926 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:05.831 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:05.831+0000 7fddb6c497c0 -1 osd.1 78 log_to_monitors true 2024-08-18T21:58:05.928 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:05.928 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:05.928 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:58:05.928 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:58:05.928 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:05.929 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-08-18T21:58:06.499 INFO:tasks.workunit.client.0.smithi099.stdout:osd.1 up in weight 1 up_from 89 up_thru 76 down_at 79 last_clean_interval [63,78) [v2:127.0.0.1:6810/2191062244,v1:127.0.0.1:6811/2191062244] [v2:127.0.0.1:6812/2191062244,v1:127.0.0.1:6813/2191062244] exists,up 000c933c-86aa-4ddf-9c31-889e86e7877b 2024-08-18T21:58:06.500 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:58:06.500 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:58:06.500 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:58:06.500 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-08-18T21:58:06.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=142012 2024-08-18T21:58:06.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-08-18T21:58:06.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:58:06.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:58:06.502 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:58:06.503 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:58:06.504 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:58:06.505 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:58:06.505 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:58:06.505 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:58:06.505 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:58:06.505 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:58:06.505 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:58:06.505 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:58:06.505 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:58:06.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:58:06.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:58:06.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:58:06.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:58:06.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:58:06.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:58:06.506 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-08-18T21:58:06.506 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.2 2024-08-18T21:58:06.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-08-18T21:58:06.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:58:06.507 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-08-18T21:58:06.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-08-18T21:58:06.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:58:06.507 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:58:06.508 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:58:06.537 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:06.535+0000 7fca7470f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:58:06.618 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:06.615+0000 7fca7470f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:58:06.619 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:06.619+0000 7fca7470f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:58:06.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-08-18T21:58:06.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:58:06.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-08-18T21:58:06.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:58:06.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:58:06.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:06.984 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:58:06.985 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:58:06.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:06.985 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:58:07.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:08.113 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:08.111+0000 7fca7470f7c0 -1 Falling back to public interface 2024-08-18T21:58:08.520 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:08.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:08.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:58:08.521 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:58:08.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:08.521 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:58:08.960 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:09.962 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:09.962 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:09.962 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:58:09.963 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:58:09.963 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:09.963 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:58:10.040 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:10.039+0000 7fca7470f7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:58:10.385 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:10.617 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:10.615+0000 7fca7470f7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:58:11.347 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:11.347+0000 7fca7470f7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:58:11.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:11.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:11.387 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:58:11.387 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:58:11.388 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:11.388 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:58:11.809 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:11.899 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:11.899+0000 7fca7470f7c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:58:12.334 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:12.335+0000 7fca7470f7c0 -1 osd.2 81 log_to_monitors true 2024-08-18T21:58:12.811 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:12.811 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:12.811 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:58:12.812 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:58:12.812 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:12.812 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-08-18T21:58:13.363 INFO:tasks.workunit.client.0.smithi099.stdout:osd.2 up in weight 1 up_from 93 up_thru 79 down_at 82 last_clean_interval [67,81) [v2:127.0.0.1:6818/2950065415,v1:127.0.0.1:6819/2950065415] [v2:127.0.0.1:6820/2950065415,v1:127.0.0.1:6821/2950065415] exists,up dab4835d-5f3e-4530-8813-db3b51c8e6fa 2024-08-18T21:58:13.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:58:13.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:58:13.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:58:13.364 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-08-18T21:58:13.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=142965 2024-08-18T21:58:13.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-08-18T21:58:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:58:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T21:58:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-08-18T21:58:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T21:58:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-08-18T21:58:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-08-18T21:58:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:58:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:58:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:58:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-08-18T21:58:13.368 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:58:13.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:58:13.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:58:13.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:58:13.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:58:13.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:58:13.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:58:13.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T21:58:13.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T21:58:13.370 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-08-18T21:58:13.371 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-08-18T21:58:13.371 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.3 2024-08-18T21:58:13.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:58:13.372 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-08-18T21:58:13.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-08-18T21:58:13.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T21:58:13.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T21:58:13.372 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T21:58:13.399 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:13.399+0000 7f3a863a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:58:13.401 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:13.399+0000 7f3a863a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:58:13.402 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:13.399+0000 7f3a863a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:58:14.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-08-18T21:58:14.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:58:14.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-08-18T21:58:14.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:58:14.096 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:58:14.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:14.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:58:14.097 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:58:14.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:14.097 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:58:14.603 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:14.953 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:14.950+0000 7f3a863a17c0 -1 Falling back to public interface 2024-08-18T21:58:15.605 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:15.605 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:58:15.605 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:15.605 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:58:15.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:15.606 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:58:16.020 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:16.830 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:16.826+0000 7f3a863a17c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:58:17.022 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:17.022 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:17.023 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:58:17.023 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:58:17.023 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:17.023 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:58:17.371 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:17.370+0000 7f3a863a17c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:58:17.458 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:18.135 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:18.134+0000 7f3a863a17c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:58:18.459 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:18.459 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:18.459 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:58:18.459 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:58:18.460 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:18.460 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:58:18.681 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:18.678+0000 7f3a863a17c0 -1 bluefs can not change current alloc_size 0x10000 to new alloc_size 0x100000 2024-08-18T21:58:18.987 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:58:19.336 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:58:19.334+0000 7f3a863a17c0 -1 osd.3 83 log_to_monitors true 2024-08-18T21:58:19.989 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:58:19.989 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:58:19.989 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:58:19.990 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:58:19.990 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:58:19.990 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-08-18T21:58:20.430 INFO:tasks.workunit.client.0.smithi099.stdout:osd.3 up in weight 1 up_from 98 up_thru 98 down_at 84 last_clean_interval [72,83) [v2:127.0.0.1:6826/478591815,v1:127.0.0.1:6827/478591815] [v2:127.0.0.1:6828/478591815,v1:127.0.0.1:6829/478591815] exists,up fc8c4dc2-5d7e-447d-b7fe-0621f00c25c1 2024-08-18T21:58:20.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:58:20.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:58:20.430 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:58:20.431 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-08-18T21:58:20.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=143928 2024-08-18T21:58:20.433 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-08-18T21:58:20.719 INFO:tasks.workunit.client.0.smithi099.stdout:hints = 1 2024-08-18T21:58:20.719 INFO:tasks.workunit.client.0.smithi099.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-08-18T21:58:20.719 INFO:tasks.workunit.client.0.smithi099.stdout:Object prefix: benchmark_data_smithi099_144886 2024-08-18T21:58:20.719 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T21:58:20.720 INFO:tasks.workunit.client.0.smithi099.stdout: 0 0 0 0 0 0 - 0 2024-08-18T21:58:21.719 INFO:tasks.workunit.client.0.smithi099.stdout: 1 16 68 52 0.203103 0.203125 0.183465 0.287051 2024-08-18T21:58:22.719 INFO:tasks.workunit.client.0.smithi099.stdout: 2 16 172 156 0.304649 0.40625 0.125265 0.194602 2024-08-18T21:58:23.719 INFO:tasks.workunit.client.0.smithi099.stdout: 3 16 252 236 0.307255 0.3125 0.168066 0.196649 2024-08-18T21:58:24.720 INFO:tasks.workunit.client.0.smithi099.stdout: 4 16 354 338 0.330034 0.398438 0.155567 0.184795 2024-08-18T21:58:25.720 INFO:tasks.workunit.client.0.smithi099.stdout: 5 16 463 447 0.349171 0.425781 0.171561 0.177815 2024-08-18T21:58:26.720 INFO:tasks.workunit.client.0.smithi099.stdout: 6 16 562 546 0.355423 0.386719 0.221419 0.173057 2024-08-18T21:58:27.720 INFO:tasks.workunit.client.0.smithi099.stdout: 7 15 650 635 0.354307 0.347656 0.178833 0.174918 2024-08-18T21:58:28.720 INFO:tasks.workunit.client.0.smithi099.stdout: 8 16 738 722 0.352491 0.339844 0.157245 0.175242 2024-08-18T21:58:29.720 INFO:tasks.workunit.client.0.smithi099.stdout: 9 16 849 833 0.361497 0.433594 0.137923 0.171854 2024-08-18T21:58:30.721 INFO:tasks.workunit.client.0.smithi099.stdout: 10 16 964 948 0.370264 0.449219 0.174367 0.168115 2024-08-18T21:58:31.721 INFO:tasks.workunit.client.0.smithi099.stdout: 11 16 1063 1047 0.371756 0.386719 0.124689 0.167099 2024-08-18T21:58:32.721 INFO:tasks.workunit.client.0.smithi099.stdout: 12 16 1169 1153 0.375277 0.414062 0.115215 0.164854 2024-08-18T21:58:33.721 INFO:tasks.workunit.client.0.smithi099.stdout: 13 16 1240 1224 0.367739 0.277344 0.340714 0.16686 2024-08-18T21:58:34.721 INFO:tasks.workunit.client.0.smithi099.stdout: 14 16 1352 1336 0.372718 0.4375 0.162426 0.16675 2024-08-18T21:58:35.721 INFO:tasks.workunit.client.0.smithi099.stdout: 15 16 1456 1440 0.374949 0.40625 0.0799507 0.165995 2024-08-18T21:58:36.721 INFO:tasks.workunit.client.0.smithi099.stdout: 16 16 1556 1540 0.375926 0.390625 0.154818 0.165199 2024-08-18T21:58:37.721 INFO:tasks.workunit.client.0.smithi099.stdout: 17 16 1664 1648 0.378625 0.421875 0.129358 0.164224 2024-08-18T21:58:38.722 INFO:tasks.workunit.client.0.smithi099.stdout: 18 15 1746 1731 0.375597 0.324219 0.249345 0.165351 2024-08-18T21:58:39.722 INFO:tasks.workunit.client.0.smithi099.stdout: 19 16 1817 1801 0.370218 0.273438 0.125962 0.167739 2024-08-18T21:58:40.722 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T21:58:40.725096+0000 min lat: 0.078574 max lat: 0.750126 avg lat: 0.165857 2024-08-18T21:58:40.722 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T21:58:40.722 INFO:tasks.workunit.client.0.smithi099.stdout: 20 16 1938 1922 0.375336 0.472656 0.161702 0.165857 2024-08-18T21:58:41.722 INFO:tasks.workunit.client.0.smithi099.stdout: 21 16 2050 2034 0.378292 0.4375 0.122403 0.164681 2024-08-18T21:58:42.723 INFO:tasks.workunit.client.0.smithi099.stdout: 22 16 2158 2142 0.38027 0.421875 0.139095 0.163687 2024-08-18T21:58:43.723 INFO:tasks.workunit.client.0.smithi099.stdout: 23 16 2261 2245 0.381228 0.402344 0.181274 0.163424 2024-08-18T21:58:44.723 INFO:tasks.workunit.client.0.smithi099.stdout: 24 16 2350 2334 0.379827 0.347656 0.205646 0.162406 2024-08-18T21:58:45.723 INFO:tasks.workunit.client.0.smithi099.stdout: 25 16 2440 2424 0.378695 0.351562 0.155169 0.164305 2024-08-18T21:58:46.723 INFO:tasks.workunit.client.0.smithi099.stdout: 26 16 2566 2550 0.383058 0.492188 0.112454 0.16272 2024-08-18T21:58:47.723 INFO:tasks.workunit.client.0.smithi099.stdout: 27 16 2669 2653 0.383771 0.402344 0.133825 0.1624 2024-08-18T21:58:48.726 INFO:tasks.workunit.client.0.smithi099.stdout: 28 16 2769 2753 0.384014 0.390625 0.183555 0.162039 2024-08-18T21:58:49.723 INFO:tasks.workunit.client.0.smithi099.stdout: 29 16 2878 2862 0.385452 0.425781 0.135236 0.161782 2024-08-18T21:58:50.724 INFO:tasks.workunit.client.0.smithi099.stdout: 30 16 2980 2964 0.385882 0.398438 0.187016 0.16146 2024-08-18T21:58:51.724 INFO:tasks.workunit.client.0.smithi099.stdout:Total time run: 30.171 2024-08-18T21:58:51.724 INFO:tasks.workunit.client.0.smithi099.stdout:Total writes made: 2980 2024-08-18T21:58:51.724 INFO:tasks.workunit.client.0.smithi099.stdout:Write size: 4096 2024-08-18T21:58:51.724 INFO:tasks.workunit.client.0.smithi099.stdout:Object size: 4096 2024-08-18T21:58:51.724 INFO:tasks.workunit.client.0.smithi099.stdout:Bandwidth (MB/sec): 0.385822 2024-08-18T21:58:51.724 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Bandwidth: 0.0621403 2024-08-18T21:58:51.724 INFO:tasks.workunit.client.0.smithi099.stdout:Max bandwidth (MB/sec): 0.492188 2024-08-18T21:58:51.725 INFO:tasks.workunit.client.0.smithi099.stdout:Min bandwidth (MB/sec): 0.203125 2024-08-18T21:58:51.725 INFO:tasks.workunit.client.0.smithi099.stdout:Average IOPS: 98 2024-08-18T21:58:51.725 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev IOPS: 15.9079 2024-08-18T21:58:51.725 INFO:tasks.workunit.client.0.smithi099.stdout:Max IOPS: 126 2024-08-18T21:58:51.725 INFO:tasks.workunit.client.0.smithi099.stdout:Min IOPS: 52 2024-08-18T21:58:51.725 INFO:tasks.workunit.client.0.smithi099.stdout:Average Latency(s): 0.161756 2024-08-18T21:58:51.725 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Latency(s): 0.0602216 2024-08-18T21:58:51.725 INFO:tasks.workunit.client.0.smithi099.stdout:Max latency(s): 0.750126 2024-08-18T21:58:51.725 INFO:tasks.workunit.client.0.smithi099.stdout:Min latency(s): 0.078574 2024-08-18T21:58:51.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-08-18T21:58:51.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T21:58:51.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T21:58:51.787 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T21:58:51.787 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-18T21:58:51.788 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T21:58:51.788 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T21:58:51.788 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T21:58:51.788 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T21:58:51.788 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T21:58:51.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-18T21:58:51.908 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T21:58:51.908 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T21:58:51.908 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T21:58:51.908 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T21:58:51.908 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T21:58:52.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids='0 2024-08-18T21:58:52.350 INFO:tasks.workunit.client.0.smithi099.stderr:1 2024-08-18T21:58:52.350 INFO:tasks.workunit.client.0.smithi099.stderr:2 2024-08-18T21:58:52.350 INFO:tasks.workunit.client.0.smithi099.stderr:3' 2024-08-18T21:58:52.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T21:58:52.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:58:52.351 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T21:58:52.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=369367187469 2024-08-18T21:58:52.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 369367187469 2024-08-18T21:58:52.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-369367187469' 2024-08-18T21:58:52.534 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:58:52.535 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-08-18T21:58:52.744 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=382252089356 2024-08-18T21:58:52.744 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 382252089356 2024-08-18T21:58:52.744 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-369367187469 1-382252089356' 2024-08-18T21:58:52.745 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:58:52.745 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-08-18T21:58:52.891 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=399431958539 2024-08-18T21:58:52.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 399431958539 2024-08-18T21:58:52.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-399431958539' 2024-08-18T21:58:52.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T21:58:52.892 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-08-18T21:58:53.032 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=420906795017 2024-08-18T21:58:53.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 420906795017 2024-08-18T21:58:53.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-399431958539 3-420906795017' 2024-08-18T21:58:53.033 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:58:53.033 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-369367187469 2024-08-18T21:58:53.033 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:58:53.035 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T21:58:53.036 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-369367187469 2024-08-18T21:58:53.036 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:58:53.037 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 369367187469 2024-08-18T21:58:53.037 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=369367187469 2024-08-18T21:58:53.037 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 369367187469' 2024-08-18T21:58:53.037 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:58:53.471 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 369367187467 -lt 369367187469 2024-08-18T21:58:53.471 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T21:58:54.473 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T21:58:54.474 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T21:58:54.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 369367187469 -lt 369367187469 2024-08-18T21:58:54.923 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:58:54.924 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-382252089356 2024-08-18T21:58:54.924 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:58:54.925 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=1 2024-08-18T21:58:54.926 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 1-382252089356 2024-08-18T21:58:54.926 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:58:54.927 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=382252089356 2024-08-18T21:58:54.927 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.1 seq 382252089356' 2024-08-18T21:58:54.927 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.1 seq 382252089356 2024-08-18T21:58:54.928 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 1 2024-08-18T21:58:55.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 382252089356 -lt 382252089356 2024-08-18T21:58:55.393 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:58:55.394 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-399431958539 2024-08-18T21:58:55.395 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:58:55.396 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=2 2024-08-18T21:58:55.397 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 2-399431958539 2024-08-18T21:58:55.397 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:58:55.398 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=399431958539 2024-08-18T21:58:55.398 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.2 seq 399431958539' 2024-08-18T21:58:55.399 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.2 seq 399431958539 2024-08-18T21:58:55.399 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 2 2024-08-18T21:58:55.836 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 399431958539 -lt 399431958539 2024-08-18T21:58:55.837 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T21:58:55.837 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-420906795017 2024-08-18T21:58:55.838 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T21:58:55.839 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=3 2024-08-18T21:58:55.840 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 3-420906795017 2024-08-18T21:58:55.840 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T21:58:55.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=420906795017 2024-08-18T21:58:55.841 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.3 seq 420906795017' 2024-08-18T21:58:55.842 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.3 seq 420906795017 2024-08-18T21:58:55.842 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 3 2024-08-18T21:58:56.277 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 420906795017 -lt 420906795017 2024-08-18T21:58:56.278 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T21:58:56.278 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:58:56.278 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:58:56.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-08-18T21:58:56.868 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T21:58:56.869 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T21:58:56.869 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T21:58:56.869 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T21:58:56.869 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T21:58:56.869 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T21:58:56.870 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T21:58:57.319 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-08-18T21:58:57.320 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T21:58:57.320 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T21:58:57.320 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T21:58:57.872 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-08-18T21:58:57.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T21:58:57.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T21:58:57.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-08-18T21:58:57.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:58:57.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:58:57.873 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-18T21:58:57.874 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:58:57.874 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:58:57.874 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:58:57.874 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:58:57.874 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:58:58.183 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:58:58.184 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:58:58.185 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:58:58.186 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:58:58.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:58:58.187 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:58:58.188 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:58:58.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:58:58.190 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:58:58.191 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:58:58.191 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:58:58.192 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:58:58.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:58:58.193 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-18T21:58:58.644 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:58:58.644 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:58:58.644 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:58:58.645 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97620 2024-08-18T21:58:58.647 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:58:58.647 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:58:58.647 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-08-18T21:58:58.647 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-08-18T21:58:58.647 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:58:58.648 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-08-18T21:58:58.648 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:58:58.648 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T21:58:58.648 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-18T21:58:58.649 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T21:58:58.649 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T21:58:58.649 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T21:58:58.649 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T21:58:58.649 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T21:58:58.652 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T21:58:58.652 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T21:58:58.653 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T21:58:58.653 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T21:58:58.654 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T21:58:58.654 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T21:58:58.655 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T21:58:58.656 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:58:58.656 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T21:58:58.656 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T21:58:58.656 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T21:58:58.657 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T21:58:58.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T21:58:58.658 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-18T21:58:58.660 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T21:58:58.660 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:58:58.660 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:58:58.660 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97620 2024-08-18T21:58:58.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T21:58:58.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T21:58:58.661 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-08-18T21:58:58.663 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-18T21:58:58.663 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:58:58.663 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:58:58.663 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.97620 2024-08-18T21:58:58.663 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-18T21:58:58.664 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-18T21:58:58.664 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-18T21:58:58.665 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-08-18T21:58:58.665 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-08-18T21:58:58.665 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:58:58.665 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-08-18T21:58:58.666 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-08-18T21:58:58.666 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-08-18T21:58:58.666 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-08-18T21:58:58.666 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-08-18T21:58:58.666 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-08-18T21:58:58.666 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-08-18T21:58:58.668 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=142f9e91-da44-40cb-9375-0342a0b8d154 --auth-supported=none ' 2024-08-18T21:58:58.668 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-08-18T21:58:58.668 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-08-18T21:58:58.668 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-08-18T21:58:58.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-08-18T21:58:58.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-08-18T21:58:58.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-08-18T21:58:58.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-08-18T21:58:58.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-08-18T21:58:58.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-08-18T21:58:58.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-08-18T21:58:58.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:58:58.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-18T21:58:58.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-18T21:58:58.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-18T21:58:58.670 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-08-18T21:58:58.670 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-08-18T21:58:59.146 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-18T21:58:59.146 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:58:59.146 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:58:59.147 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:58:59.147 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:58:59.147 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:58:59.147 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:58:59.148 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-18T21:58:59.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-18T21:58:59.376 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-18T21:58:59.376 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:58:59.377 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:58:59.377 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-18T21:58:59.377 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-18T21:58:59.378 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:58:59.378 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:58:59.378 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:58:59.379 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:58:59.379 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:58:59.379 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:58:59.379 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97620/ceph-mon.a.asok 2024-08-18T21:58:59.379 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:58:59.379 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97620/ceph-mon.a.asok config get fsid 2024-08-18T21:58:59.495 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-18T21:58:59.495 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T21:58:59.495 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T21:58:59.495 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-18T21:58:59.495 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-18T21:58:59.496 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T21:58:59.496 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T21:58:59.496 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T21:58:59.496 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T21:58:59.496 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:58:59.496 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:58:59.496 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97620/ceph-mon.a.asok 2024-08-18T21:58:59.497 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T21:58:59.497 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97620/ceph-mon.a.asok config get mon_host 2024-08-18T21:58:59.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-08-18T21:58:59.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:58:59.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-18T21:58:59.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-18T21:58:59.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-18T21:58:59.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-08-18T21:58:59.600 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-18T21:59:00.511 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-18T21:59:00.511 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:59:00.511 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:59:00.511 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:59:00.511 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:59:00.511 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:59:00.512 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:59:00.512 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:59:00.513 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T21:59:00.555 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-08-18T21:59:00.555 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T21:59:00.555 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T21:59:00.555 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-18T21:59:00.555 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T21:59:00.555 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-18T21:59:00.555 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=142f9e91-da44-40cb-9375-0342a0b8d154 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-08-18T21:59:00.555 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T21:59:00.556 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T21:59:00.557 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T21:59:00.558 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-18T21:59:00.558 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T21:59:00.558 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=93dc2878-4fad-4688-8f07-842eda48fe1d 2024-08-18T21:59:00.558 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 93dc2878-4fad-4688-8f07-842eda48fe1d' 2024-08-18T21:59:00.559 INFO:tasks.workunit.client.0.smithi099.stdout:add osd0 93dc2878-4fad-4688-8f07-842eda48fe1d 2024-08-18T21:59:00.559 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T21:59:00.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCkbsJmLvaMIhAAudb6ypKspdHKAmK0DUBRNw== 2024-08-18T21:59:00.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCkbsJmLvaMIhAAudb6ypKspdHKAmK0DUBRNw=="}' 2024-08-18T21:59:00.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 93dc2878-4fad-4688-8f07-842eda48fe1d -i td/osd-bluefs-volume-ops/0/new.json 2024-08-18T21:59:00.885 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:59:00.901 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-08-18T21:59:00.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=142f9e91-da44-40cb-9375-0342a0b8d154 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCkbsJmLvaMIhAAudb6ypKspdHKAmK0DUBRNw== --osd-uuid 93dc2878-4fad-4688-8f07-842eda48fe1d 2024-08-18T21:59:00.941 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:59:00.937+0000 7f02cf1e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:59:00.963 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:59:00.961+0000 7f02cf1e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:59:00.965 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:59:00.961+0000 7f02cf1e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:59:00.965 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:59:00.961+0000 7f02cf1e07c0 -1 bdev(0x55a81c8c4800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-08-18T21:59:00.965 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:59:00.961+0000 7f02cf1e07c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-08-18T21:59:09.287 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-08-18T21:59:09.287 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T21:59:09.288 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-18T21:59:09.288 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd0 key to auth repository 2024-08-18T21:59:09.289 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T21:59:09.870 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T21:59:09.871 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-18T21:59:09.871 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=142f9e91-da44-40cb-9375-0342a0b8d154 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T21:59:09.871 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T21:59:09.871 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T21:59:09.871 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T21:59:09.907 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:59:09.909+0000 7fa91867d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:59:09.924 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:59:09.925+0000 7fa91867d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:59:09.925 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:59:09.925+0000 7fa91867d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T21:59:10.420 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-18T21:59:10.420 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T21:59:10.420 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T21:59:10.420 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T21:59:10.420 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T21:59:10.420 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:59:10.421 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T21:59:10.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T21:59:10.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:59:10.421 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:59:10.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:59:11.481 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:59:11.481+0000 7fa91867d7c0 -1 Falling back to public interface 2024-08-18T21:59:11.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:59:11.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:59:11.888 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T21:59:11.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T21:59:11.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:59:11.888 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:59:12.330 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:59:13.331 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:59:13.332 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T21:59:13.332 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:59:13.332 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T21:59:13.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:59:13.333 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:59:13.749 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:59:14.752 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:59:14.752 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:59:14.752 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T21:59:14.752 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T21:59:14.752 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:59:14.753 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:59:15.179 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:59:15.680 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T21:59:15.681+0000 7fa91867d7c0 -1 osd.0 0 log_to_monitors true 2024-08-18T21:59:16.181 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:59:16.181 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:59:16.181 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T21:59:16.182 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T21:59:16.182 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:59:16.182 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:59:17.361 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:59:18.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:59:18.363 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:59:18.363 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T21:59:18.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T21:59:18.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:59:18.364 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:59:19.586 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T21:59:20.587 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T21:59:20.588 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T21:59:20.588 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T21:59:20.588 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T21:59:20.588 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T21:59:20.588 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T21:59:21.075 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1608188475,v1:127.0.0.1:6803/1608188475] [v2:127.0.0.1:6804/1608188475,v1:127.0.0.1:6805/1608188475] exists,up 93dc2878-4fad-4688-8f07-842eda48fe1d 2024-08-18T21:59:21.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T21:59:21.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T21:59:21.076 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T21:59:21.076 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-18T21:59:21.077 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=146784 2024-08-18T21:59:21.077 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-08-18T21:59:26.079 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-08-18T21:59:26.079 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-08-18T21:59:26.734 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'foo' already exists 2024-08-18T21:59:26.755 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-18T21:59:27.756 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-08-18T21:59:27.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-08-18T21:59:27.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-08-18T21:59:27.757 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-08-18T21:59:27.879 INFO:tasks.workunit.client.0.smithi099.stdout:hints = 1 2024-08-18T21:59:27.881 INFO:tasks.workunit.client.0.smithi099.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-08-18T21:59:27.881 INFO:tasks.workunit.client.0.smithi099.stdout:Object prefix: benchmark_data_smithi099_148076 2024-08-18T21:59:27.924 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T21:59:27.924 INFO:tasks.workunit.client.0.smithi099.stdout: 0 0 0 0 0 0 - 0 2024-08-18T21:59:28.924 INFO:tasks.workunit.client.0.smithi099.stdout: 1 16 17 1 3.99917 4 0.26066 0.26066 2024-08-18T21:59:29.924 INFO:tasks.workunit.client.0.smithi099.stdout: 2 16 33 17 33.9934 64 0.876523 0.98839 2024-08-18T21:59:30.924 INFO:tasks.workunit.client.0.smithi099.stdout: 3 16 49 33 43.9919 64 1.0443 0.988001 2024-08-18T21:59:31.925 INFO:tasks.workunit.client.0.smithi099.stdout: 4 16 65 49 48.9915 64 1.00972 1.00759 2024-08-18T21:59:32.925 INFO:tasks.workunit.client.0.smithi099.stdout: 5 16 81 65 51.9911 64 1.42012 1.0973 2024-08-18T21:59:33.925 INFO:tasks.workunit.client.0.smithi099.stdout: 6 16 97 81 53.9908 64 1.07706 1.06997 2024-08-18T21:59:34.925 INFO:tasks.workunit.client.0.smithi099.stdout: 7 16 113 97 55.419 64 1.13581 1.10093 2024-08-18T21:59:35.925 INFO:tasks.workunit.client.0.smithi099.stdout: 8 16 128 112 55.9904 60 1.2562 1.1174 2024-08-18T21:59:36.925 INFO:tasks.workunit.client.0.smithi099.stdout: 9 16 144 128 56.8794 64 1.06995 1.1112 2024-08-18T21:59:37.926 INFO:tasks.workunit.client.0.smithi099.stdout: 10 16 145 129 51.5914 4 1.01863 1.11048 2024-08-18T21:59:38.926 INFO:tasks.workunit.client.0.smithi099.stdout:Total time run: 10.3657 2024-08-18T21:59:38.926 INFO:tasks.workunit.client.0.smithi099.stdout:Total writes made: 145 2024-08-18T21:59:38.926 INFO:tasks.workunit.client.0.smithi099.stdout:Write size: 4194304 2024-08-18T21:59:38.926 INFO:tasks.workunit.client.0.smithi099.stdout:Object size: 4194304 2024-08-18T21:59:38.926 INFO:tasks.workunit.client.0.smithi099.stdout:Bandwidth (MB/sec): 55.954 2024-08-18T21:59:38.926 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Bandwidth: 25.1184 2024-08-18T21:59:38.926 INFO:tasks.workunit.client.0.smithi099.stdout:Max bandwidth (MB/sec): 64 2024-08-18T21:59:38.927 INFO:tasks.workunit.client.0.smithi099.stdout:Min bandwidth (MB/sec): 4 2024-08-18T21:59:38.927 INFO:tasks.workunit.client.0.smithi099.stdout:Average IOPS: 13 2024-08-18T21:59:38.927 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev IOPS: 6.2796 2024-08-18T21:59:38.927 INFO:tasks.workunit.client.0.smithi099.stdout:Max IOPS: 16 2024-08-18T21:59:38.927 INFO:tasks.workunit.client.0.smithi099.stdout:Min IOPS: 1 2024-08-18T21:59:38.927 INFO:tasks.workunit.client.0.smithi099.stdout:Average Latency(s): 1.13259 2024-08-18T21:59:38.927 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Latency(s): 0.163117 2024-08-18T21:59:38.927 INFO:tasks.workunit.client.0.smithi099.stdout:Max latency(s): 1.42012 2024-08-18T21:59:38.927 INFO:tasks.workunit.client.0.smithi099.stdout:Min latency(s): 0.26066 2024-08-18T21:59:38.986 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-08-18T21:59:48.989 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-08-18T21:59:48.989 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-08-18T21:59:49.166 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=984612864 2024-08-18T21:59:49.167 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-08-18T21:59:49.167 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-08-18T21:59:49.347 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=0 2024-08-18T21:59:49.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-08-18T21:59:49.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 0 -eq 0 2024-08-18T21:59:49.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ 1 -le 5 ]] 2024-08-18T21:59:49.348 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-08-18T21:59:49.381 INFO:tasks.workunit.client.0.smithi099.stdout:hints = 1 2024-08-18T21:59:49.382 INFO:tasks.workunit.client.0.smithi099.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-08-18T21:59:49.383 INFO:tasks.workunit.client.0.smithi099.stdout:Object prefix: benchmark_data_smithi099_148149 2024-08-18T21:59:49.413 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T21:59:49.413 INFO:tasks.workunit.client.0.smithi099.stdout: 0 0 0 0 0 0 - 0 2024-08-18T21:59:50.412 INFO:tasks.workunit.client.0.smithi099.stdout: 1 16 23 7 27.9961 28 0.98964 0.870259 2024-08-18T21:59:51.413 INFO:tasks.workunit.client.0.smithi099.stdout: 2 16 48 32 63.9895 100 0.798481 0.881978 2024-08-18T21:59:52.413 INFO:tasks.workunit.client.0.smithi099.stdout: 3 16 64 48 63.99 64 1.16588 0.966223 2024-08-18T21:59:53.414 INFO:tasks.workunit.client.0.smithi099.stdout: 4 16 81 65 64.99 68 0.734079 0.906263 2024-08-18T21:59:54.413 INFO:tasks.workunit.client.0.smithi099.stdout: 5 16 81 65 51.992 0 - 0.906263 2024-08-18T21:59:55.413 INFO:tasks.workunit.client.0.smithi099.stdout: 6 16 112 96 63.9898 62 0.825372 0.980803 2024-08-18T21:59:56.413 INFO:tasks.workunit.client.0.smithi099.stdout: 7 16 113 97 55.4196 4 0.859871 0.979556 2024-08-18T21:59:57.414 INFO:tasks.workunit.client.0.smithi099.stdout: 8 16 129 113 56.4908 64 1.74649 1.06541 2024-08-18T21:59:58.414 INFO:tasks.workunit.client.0.smithi099.stdout: 9 16 129 113 50.2141 0 - 1.06541 2024-08-18T21:59:59.414 INFO:tasks.workunit.client.0.smithi099.stdout: 10 16 129 113 45.1926 0 - 1.06541 2024-08-18T22:00:00.414 INFO:tasks.workunit.client.0.smithi099.stdout: 11 16 129 113 41.0842 0 - 1.06541 2024-08-18T22:00:01.415 INFO:tasks.workunit.client.0.smithi099.stdout:Total time run: 11.4837 2024-08-18T22:00:01.415 INFO:tasks.workunit.client.0.smithi099.stdout:Total writes made: 129 2024-08-18T22:00:01.415 INFO:tasks.workunit.client.0.smithi099.stdout:Write size: 4194304 2024-08-18T22:00:01.415 INFO:tasks.workunit.client.0.smithi099.stdout:Object size: 4194304 2024-08-18T22:00:01.415 INFO:tasks.workunit.client.0.smithi099.stdout:Bandwidth (MB/sec): 44.9333 2024-08-18T22:00:01.415 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Bandwidth: 36.9225 2024-08-18T22:00:01.415 INFO:tasks.workunit.client.0.smithi099.stdout:Max bandwidth (MB/sec): 100 2024-08-18T22:00:01.415 INFO:tasks.workunit.client.0.smithi099.stdout:Min bandwidth (MB/sec): 0 2024-08-18T22:00:01.415 INFO:tasks.workunit.client.0.smithi099.stdout:Average IOPS: 11 2024-08-18T22:00:01.415 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev IOPS: 9.19585 2024-08-18T22:00:01.416 INFO:tasks.workunit.client.0.smithi099.stdout:Max IOPS: 25 2024-08-18T22:00:01.416 INFO:tasks.workunit.client.0.smithi099.stdout:Min IOPS: 0 2024-08-18T22:00:01.416 INFO:tasks.workunit.client.0.smithi099.stdout:Average Latency(s): 1.41466 2024-08-18T22:00:01.416 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Latency(s): 0.977965 2024-08-18T22:00:01.416 INFO:tasks.workunit.client.0.smithi099.stdout:Max latency(s): 3.90673 2024-08-18T22:00:01.416 INFO:tasks.workunit.client.0.smithi099.stdout:Min latency(s): 0.154143 2024-08-18T22:00:01.517 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-08-18T22:00:11.521 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-08-18T22:00:11.521 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-08-18T22:00:11.700 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=90177536 2024-08-18T22:00:11.700 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-08-18T22:00:11.700 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-08-18T22:00:11.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=314114048 2024-08-18T22:00:11.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-08-18T22:00:11.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 314114048 -eq 0 2024-08-18T22:00:11.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-08-18T22:00:11.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 314114048 -gt 0 2024-08-18T22:00:11.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 146784 2024-08-18T22:00:11.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-08-18T22:00:12.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 146784 2024-08-18T22:00:12.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (146784) - No such process 2024-08-18T22:00:12.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-08-18T22:00:13.333 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already down. 2024-08-18T22:00:13.349 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-08-18T22:00:13.377 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T22:00:16.751 INFO:tasks.workunit.client.0.smithi099.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-08-18T22:00:16.778 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-08-18T22:00:16.807 INFO:tasks.workunit.client.0.smithi099.stdout:inferring bluefs devices from bluestore path 2024-08-18T22:00:18.120 INFO:tasks.workunit.client.0.smithi099.stdout:1 : device size 0xffffe000 : using 0x183a8000(388 MiB) 2024-08-18T22:00:18.133 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-08-18T22:00:20.503 INFO:tasks.workunit.client.0.smithi099.stdout:fsck success 2024-08-18T22:00:20.564 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-08-18T22:00:20.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T22:00:20.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-08-18T22:00:20.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-08-18T22:00:20.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-08-18T22:00:20.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-18T22:00:20.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=142f9e91-da44-40cb-9375-0342a0b8d154 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-08-18T22:00:20.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T22:00:20.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T22:00:20.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T22:00:20.565 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-18T22:00:20.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-18T22:00:20.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-08-18T22:00:20.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-08-18T22:00:20.566 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T22:00:20.566 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-08-18T22:00:20.566 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T22:00:20.566 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T22:00:20.566 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T22:00:20.566 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T22:00:20.566 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T22:00:20.566 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T22:00:20.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T22:00:20.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-08-18T22:00:20.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T22:00:20.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T22:00:20.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T22:00:20.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T22:00:20.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T22:00:20.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T22:00:20.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-08-18T22:00:20.568 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-08-18T22:00:20.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-18T22:00:20.569 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-08-18T22:00:20.569 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T22:00:20.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=142f9e91-da44-40cb-9375-0342a0b8d154 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T22:00:20.570 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-08-18T22:00:20.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-08-18T22:00:20.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-08-18T22:00:20.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-08-18T22:00:20.570 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-08-18T22:00:20.605 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:20.603+0000 7fd5142bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T22:00:20.608 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:20.607+0000 7fd5142bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T22:00:20.609 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:20.607+0000 7fd5142bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T22:00:21.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-08-18T22:00:21.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T22:00:21.011 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T22:00:21.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T22:00:21.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T22:00:21.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:21.012 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T22:00:21.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T22:00:21.012 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:21.013 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:21.440 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:21.814 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:21.811+0000 7fd5142bb7c0 -1 Falling back to public interface 2024-08-18T22:00:22.442 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:22.442 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:22.442 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T22:00:22.443 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T22:00:22.443 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:22.443 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:22.884 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:23.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:23.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:23.886 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T22:00:23.887 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T22:00:23.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:23.887 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:24.351 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:25.352 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:25.353 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:25.353 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T22:00:25.354 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T22:00:25.354 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:25.354 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:25.775 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:26.661 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:26.659+0000 7fd5142bb7c0 -1 osd.0 10 log_to_monitors true 2024-08-18T22:00:26.776 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:26.776 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:26.777 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T22:00:26.777 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T22:00:26.777 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:26.777 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:27.899 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:28.900 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:28.900 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:28.901 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T22:00:28.901 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T22:00:28.901 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:28.901 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:29.831 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:30.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:30.832 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:30.833 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T22:00:30.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T22:00:30.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:30.833 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:31.367 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/1112835143,v1:127.0.0.1:6803/1112835143] [v2:127.0.0.1:6804/1112835143,v1:127.0.0.1:6805/1112835143] exists,up 93dc2878-4fad-4688-8f07-842eda48fe1d 2024-08-18T22:00:31.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T22:00:31.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T22:00:31.367 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T22:00:31.367 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-18T22:00:31.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=149381 2024-08-18T22:00:31.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-08-18T22:00:31.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-08-18T22:00:31.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-08-18T22:00:31.369 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-08-18T22:00:31.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-08-18T22:00:31.370 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-08-18T22:00:31.370 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-08-18T22:00:31.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-08-18T22:00:31.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-08-18T22:00:31.370 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-08-18T22:00:31.477 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-08-18T22:00:31.477 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-08-18T22:00:31.477 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-08-18T22:00:31.477 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-08-18T22:00:31.477 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2214: flush_pg_stats: local timeout=300 2024-08-18T22:00:31.477 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ceph osd ls 2024-08-18T22:00:31.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: ids=0 2024-08-18T22:00:31.910 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: seqs= 2024-08-18T22:00:31.910 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for osd in $ids 2024-08-18T22:00:31.910 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-08-18T22:00:32.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: seq=55834574852 2024-08-18T22:00:32.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: test -z 55834574852 2024-08-18T22:00:32.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: seqs=' 0-55834574852' 2024-08-18T22:00:32.103 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: for s in $seqs 2024-08-18T22:00:32.103 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-55834574852 2024-08-18T22:00:32.103 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 1 2024-08-18T22:00:32.104 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: osd=0 2024-08-18T22:00:32.105 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 0-55834574852 2024-08-18T22:00:32.105 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: cut -d - -f 2 2024-08-18T22:00:32.106 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: seq=55834574852 2024-08-18T22:00:32.106 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-08-18T22:00:32.106 INFO:tasks.workunit.client.0.smithi099.stdout:waiting osd.0 seq 55834574852 2024-08-18T22:00:32.107 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T22:00:32.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 1 -lt 55834574852 2024-08-18T22:00:32.519 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: sleep 1 2024-08-18T22:00:33.520 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: flush_pg_stats: '[' 300 -eq 0 ']' 2024-08-18T22:00:33.521 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: ceph osd last-stat-seq 0 2024-08-18T22:00:33.978 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: test 55834574852 -lt 55834574852 2024-08-18T22:00:33.979 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-08-18T22:00:33.979 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T22:00:33.979 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T22:00:34.546 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-08-18T22:00:34.546 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-08-18T22:00:34.547 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-08-18T22:00:34.547 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-08-18T22:00:34.547 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-08-18T22:00:34.547 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-08-18T22:00:34.547 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-08-18T22:00:34.547 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-08-18T22:00:34.968 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-08-18T22:00:34.969 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-08-18T22:00:34.969 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-08-18T22:00:34.969 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-08-18T22:00:35.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-08-18T22:00:35.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-08-18T22:00:35.535 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-08-18T22:00:35.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-08-18T22:00:35.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-18T22:00:35.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T22:00:35.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-18T22:00:35.536 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T22:00:35.536 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T22:00:35.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T22:00:35.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T22:00:35.536 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T22:00:35.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T22:00:35.845 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T22:00:35.846 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T22:00:35.846 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T22:00:35.847 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T22:00:35.848 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T22:00:35.848 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T22:00:35.849 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T22:00:35.849 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T22:00:35.850 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T22:00:35.850 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T22:00:35.851 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T22:00:35.852 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T22:00:35.853 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-18T22:00:35.889 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T22:00:35.889 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T22:00:35.889 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T22:00:35.889 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97620 2024-08-18T22:00:35.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T22:00:35.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T22:00:35.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-08-18T22:00:35.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-08-18T22:00:35.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-08-18T22:00:35.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-08-18T22:00:35.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-18T22:00:35.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-08-18T22:00:35.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-18T22:00:35.893 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T22:00:35.893 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T22:00:35.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T22:00:35.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T22:00:35.893 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T22:00:35.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T22:00:35.896 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T22:00:35.897 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T22:00:35.897 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T22:00:35.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T22:00:35.898 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T22:00:35.899 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T22:00:35.899 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T22:00:35.899 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T22:00:35.900 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T22:00:35.900 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T22:00:35.901 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T22:00:35.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-08-18T22:00:35.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-18T22:00:35.904 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T22:00:35.904 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T22:00:35.904 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T22:00:35.904 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97620 2024-08-18T22:00:35.905 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T22:00:35.905 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T22:00:35.905 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-08-18T22:00:35.906 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-08-18T22:00:35.906 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T22:00:35.906 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T22:00:35.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.97620 2024-08-18T22:00:35.908 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-08-18T22:00:35.908 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-08-18T22:00:35.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-08-18T22:00:35.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-08-18T22:00:35.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-08-18T22:00:35.909 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-08-18T22:00:35.910 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-08-18T22:00:35.910 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-08-18T22:00:35.910 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-08-18T22:00:35.910 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-08-18T22:00:35.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-08-18T22:00:35.911 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-08-18T22:00:35.911 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-08-18T22:00:35.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=febfd722-958e-4093-bac7-b65660748030 --auth-supported=none ' 2024-08-18T22:00:35.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-08-18T22:00:35.912 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-08-18T22:00:35.913 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-08-18T22:00:35.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-08-18T22:00:35.914 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-08-18T22:00:36.418 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-08-18T22:00:36.418 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T22:00:36.418 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T22:00:36.418 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T22:00:36.418 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T22:00:36.418 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T22:00:36.419 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T22:00:36.419 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-08-18T22:00:36.668 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-08-18T22:00:36.669 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-08-18T22:00:36.669 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T22:00:36.669 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T22:00:36.669 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-08-18T22:00:36.671 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-08-18T22:00:36.671 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T22:00:36.671 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T22:00:36.671 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T22:00:36.671 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T22:00:36.671 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T22:00:36.672 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T22:00:36.672 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97620/ceph-mon.a.asok 2024-08-18T22:00:36.672 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T22:00:36.672 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97620/ceph-mon.a.asok config get fsid 2024-08-18T22:00:36.781 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-08-18T22:00:36.782 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-08-18T22:00:36.782 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-08-18T22:00:36.782 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-08-18T22:00:36.783 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-08-18T22:00:36.783 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-08-18T22:00:36.783 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-08-18T22:00:36.783 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-08-18T22:00:36.783 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-08-18T22:00:36.783 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T22:00:36.783 INFO:tasks.workunit.client.0.smithi099.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T22:00:36.783 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.97620/ceph-mon.a.asok 2024-08-18T22:00:36.784 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-08-18T22:00:36.784 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.97620/ceph-mon.a.asok config get mon_host 2024-08-18T22:00:36.882 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-08-18T22:00:36.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-08-18T22:00:36.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-08-18T22:00:36.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-08-18T22:00:36.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-08-18T22:00:36.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-08-18T22:00:36.883 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-08-18T22:00:37.824 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-08-18T22:00:37.824 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T22:00:37.824 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T22:00:37.824 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T22:00:37.824 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T22:00:37.824 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T22:00:37.824 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T22:00:37.825 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T22:00:37.826 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-08-18T22:00:37.901 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-08-18T22:00:37.901 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-08-18T22:00:37.901 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=142f9e91-da44-40cb-9375-0342a0b8d154 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=febfd722-958e-4093-bac7-b65660748030 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-08-18T22:00:37.902 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-08-18T22:00:37.903 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-08-18T22:00:37.903 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-08-18T22:00:37.903 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-08-18T22:00:37.906 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-08-18T22:00:37.906 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T22:00:37.906 INFO:tasks.workunit.client.0.smithi099.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T22:00:37.906 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T22:00:37.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' 2024-08-18T22:00:37.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-08-18T22:00:37.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-08-18T22:00:37.906 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-08-18T22:00:37.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-08-18T22:00:37.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-08-18T22:00:37.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-08-18T22:00:37.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-08-18T22:00:37.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-08-18T22:00:37.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-08-18T22:00:37.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-08-18T22:00:37.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-08-18T22:00:37.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-08-18T22:00:37.907 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-08-18T22:00:37.907 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ff3cc74-db3f-4f0d-92bf-b15d9cbcb5e3 2024-08-18T22:00:37.908 INFO:tasks.workunit.client.0.smithi099.stdout:add osd0 7ff3cc74-db3f-4f0d-92bf-b15d9cbcb5e3 2024-08-18T22:00:37.908 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7ff3cc74-db3f-4f0d-92bf-b15d9cbcb5e3' 2024-08-18T22:00:37.908 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-08-18T22:00:37.933 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFb8JmgzSrNxAAyFrZn5KkmKKIwn/v5jZaIw== 2024-08-18T22:00:37.933 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFb8JmgzSrNxAAyFrZn5KkmKKIwn/v5jZaIw=="}' 2024-08-18T22:00:37.933 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ff3cc74-db3f-4f0d-92bf-b15d9cbcb5e3 -i td/osd-bluefs-volume-ops/0/new.json 2024-08-18T22:00:38.265 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T22:00:38.280 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-08-18T22:00:38.282 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=142f9e91-da44-40cb-9375-0342a0b8d154 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=febfd722-958e-4093-bac7-b65660748030 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFb8JmgzSrNxAAyFrZn5KkmKKIwn/v5jZaIw== --osd-uuid 7ff3cc74-db3f-4f0d-92bf-b15d9cbcb5e3 2024-08-18T22:00:38.334 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:38.331+0000 7f3d978fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T22:00:38.341 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:38.339+0000 7f3d978fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T22:00:38.342 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:38.339+0000 7f3d978fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T22:00:38.343 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:38.343+0000 7f3d978fc7c0 -1 bdev(0x5617b1108800 td/osd-bluefs-volume-ops/0/block) open stat got: (1) Operation not permitted 2024-08-18T22:00:38.343 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:38.343+0000 7f3d978fc7c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-08-18T22:00:46.493 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-08-18T22:00:46.493 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-08-18T22:00:46.494 INFO:tasks.workunit.client.0.smithi099.stdout:adding osd0 key to auth repository 2024-08-18T22:00:46.494 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-08-18T22:00:46.494 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-08-18T22:00:47.066 INFO:tasks.workunit.client.0.smithi099.stdout:start osd.0 2024-08-18T22:00:47.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-08-18T22:00:47.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3a41e8bd-3fee-45d8-b9c3-0e2bd10dbf1b --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=142f9e91-da44-40cb-9375-0342a0b8d154 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=febfd722-958e-4093-bac7-b65660748030 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.97620/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-08-18T22:00:47.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-08-18T22:00:47.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-08-18T22:00:47.067 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-08-18T22:00:47.110 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:47.106+0000 7f8812a0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T22:00:47.137 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:47.134+0000 7f8812a0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T22:00:47.138 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:47.134+0000 7f8812a0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-08-18T22:00:47.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-08-18T22:00:47.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-08-18T22:00:47.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-08-18T22:00:47.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-08-18T22:00:47.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-08-18T22:00:47.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:47.540 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-08-18T22:00:47.541 INFO:tasks.workunit.client.0.smithi099.stdout:0 2024-08-18T22:00:47.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:47.541 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:47.995 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:48.373 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:48.370+0000 7f8812a0a7c0 -1 Falling back to public interface 2024-08-18T22:00:48.997 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:48.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:48.998 INFO:tasks.workunit.client.0.smithi099.stdout:1 2024-08-18T22:00:48.998 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-08-18T22:00:48.999 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:48.999 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:49.444 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:50.446 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:50.446 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:50.446 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-08-18T22:00:50.446 INFO:tasks.workunit.client.0.smithi099.stdout:2 2024-08-18T22:00:50.447 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:50.447 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:50.892 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:51.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:51.894 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:51.894 INFO:tasks.workunit.client.0.smithi099.stdout:3 2024-08-18T22:00:51.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-08-18T22:00:51.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:51.895 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:52.350 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:53.022 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:00:53.018+0000 7f8812a0a7c0 -1 osd.0 0 log_to_monitors true 2024-08-18T22:00:53.351 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:53.352 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:53.352 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-08-18T22:00:53.352 INFO:tasks.workunit.client.0.smithi099.stdout:4 2024-08-18T22:00:53.352 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:53.352 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:54.736 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:55.737 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:55.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:55.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-08-18T22:00:55.738 INFO:tasks.workunit.client.0.smithi099.stdout:5 2024-08-18T22:00:55.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:55.738 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:56.856 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:57.858 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:57.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:57.859 INFO:tasks.workunit.client.0.smithi099.stdout:6 2024-08-18T22:00:57.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-08-18T22:00:57.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:57.859 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:58.359 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-08-18T22:00:59.361 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-08-18T22:00:59.361 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-08-18T22:00:59.362 INFO:tasks.workunit.client.0.smithi099.stdout:7 2024-08-18T22:00:59.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-08-18T22:00:59.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-08-18T22:00:59.362 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-08-18T22:00:59.800 INFO:tasks.workunit.client.0.smithi099.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/94644402,v1:127.0.0.1:6803/94644402] [v2:127.0.0.1:6804/94644402,v1:127.0.0.1:6805/94644402] exists,up 7ff3cc74-db3f-4f0d-92bf-b15d9cbcb5e3 2024-08-18T22:00:59.800 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-08-18T22:00:59.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-08-18T22:00:59.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-08-18T22:00:59.801 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-08-18T22:00:59.801 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=152088 2024-08-18T22:00:59.802 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-08-18T22:01:04.803 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-08-18T22:01:04.804 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-08-18T22:01:05.357 INFO:tasks.workunit.client.0.smithi099.stderr:pool 'foo' already exists 2024-08-18T22:01:05.373 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-08-18T22:01:06.375 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-08-18T22:01:06.426 INFO:tasks.workunit.client.0.smithi099.stdout:hints = 1 2024-08-18T22:01:06.426 INFO:tasks.workunit.client.0.smithi099.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-08-18T22:01:06.426 INFO:tasks.workunit.client.0.smithi099.stdout:Object prefix: benchmark_data_smithi099_153148 2024-08-18T22:01:06.426 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T22:01:06.426 INFO:tasks.workunit.client.0.smithi099.stdout: 0 0 0 0 0 0 - 0 2024-08-18T22:01:07.426 INFO:tasks.workunit.client.0.smithi099.stdout: 1 16 96 80 0.312483 0.3125 0.126027 0.19508 2024-08-18T22:01:08.426 INFO:tasks.workunit.client.0.smithi099.stdout: 2 16 210 194 0.37886 0.445312 0.125517 0.158279 2024-08-18T22:01:09.426 INFO:tasks.workunit.client.0.smithi099.stdout: 3 16 306 290 0.377553 0.375 0.133832 0.159447 2024-08-18T22:01:10.426 INFO:tasks.workunit.client.0.smithi099.stdout: 4 16 432 416 0.406194 0.492188 0.118044 0.153313 2024-08-18T22:01:11.426 INFO:tasks.workunit.client.0.smithi099.stdout: 5 16 528 512 0.399944 0.375 0.118371 0.15516 2024-08-18T22:01:12.426 INFO:tasks.workunit.client.0.smithi099.stdout: 6 16 642 626 0.407495 0.445312 0.14229 0.150868 2024-08-18T22:01:13.426 INFO:tasks.workunit.client.0.smithi099.stdout: 7 16 752 736 0.410659 0.429688 0.142414 0.151942 2024-08-18T22:01:14.427 INFO:tasks.workunit.client.0.smithi099.stdout: 8 16 864 848 0.414006 0.4375 0.150562 0.150647 2024-08-18T22:01:15.427 INFO:tasks.workunit.client.0.smithi099.stdout: 9 16 960 944 0.409666 0.375 0.151386 0.151355 2024-08-18T22:01:16.427 INFO:tasks.workunit.client.0.smithi099.stdout: 10 16 1074 1058 0.413223 0.445312 0.142153 0.15011 2024-08-18T22:01:17.427 INFO:tasks.workunit.client.0.smithi099.stdout: 11 16 1184 1168 0.414715 0.429688 0.143166 0.150286 2024-08-18T22:01:18.427 INFO:tasks.workunit.client.0.smithi099.stdout: 12 16 1311 1295 0.42149 0.496094 0.125788 0.148056 2024-08-18T22:01:19.427 INFO:tasks.workunit.client.0.smithi099.stdout: 13 16 1408 1392 0.418211 0.378906 0.125562 0.149176 2024-08-18T22:01:20.428 INFO:tasks.workunit.client.0.smithi099.stdout: 14 16 1520 1504 0.419585 0.4375 0.116981 0.148838 2024-08-18T22:01:21.428 INFO:tasks.workunit.client.0.smithi099.stdout: 15 16 1616 1600 0.416608 0.375 0.15045 0.149519 2024-08-18T22:01:22.428 INFO:tasks.workunit.client.0.smithi099.stdout: 16 16 1728 1712 0.417909 0.4375 0.13355 0.148971 2024-08-18T22:01:23.428 INFO:tasks.workunit.client.0.smithi099.stdout: 17 16 1824 1808 0.415382 0.375 0.176358 0.149628 2024-08-18T22:01:24.428 INFO:tasks.workunit.client.0.smithi099.stdout: 18 16 1890 1874 0.406626 0.257812 0.150085 0.153012 2024-08-18T22:01:25.428 INFO:tasks.workunit.client.0.smithi099.stdout: 19 16 1986 1970 0.404958 0.375 0.150653 0.153181 2024-08-18T22:01:26.428 INFO:tasks.workunit.client.0.smithi099.stdout:2024-08-18T22:01:26.431112+0000 min lat: 0.0990044 max lat: 0.610078 avg lat: 0.15237 2024-08-18T22:01:26.429 INFO:tasks.workunit.client.0.smithi099.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-08-18T22:01:26.429 INFO:tasks.workunit.client.0.smithi099.stdout: 20 16 2112 2096 0.409316 0.492188 0.158401 0.15237 2024-08-18T22:01:27.429 INFO:tasks.workunit.client.0.smithi099.stdout: 21 16 2208 2192 0.407679 0.375 0.183929 0.152721 2024-08-18T22:01:28.429 INFO:tasks.workunit.client.0.smithi099.stdout: 22 16 2322 2306 0.409386 0.445312 0.116886 0.152118 2024-08-18T22:01:29.429 INFO:tasks.workunit.client.0.smithi099.stdout: 23 16 2418 2402 0.407889 0.375 0.125598 0.152454 2024-08-18T22:01:30.429 INFO:tasks.workunit.client.0.smithi099.stdout: 24 16 2546 2530 0.411724 0.5 0.133772 0.151258 2024-08-18T22:01:31.429 INFO:tasks.workunit.client.0.smithi099.stdout: 25 16 2642 2626 0.410253 0.375 0.150587 0.151546 2024-08-18T22:01:32.429 INFO:tasks.workunit.client.0.smithi099.stdout: 26 16 2768 2752 0.413401 0.492188 0.158885 0.150723 2024-08-18T22:01:33.430 INFO:tasks.workunit.client.0.smithi099.stdout: 27 16 2866 2850 0.412266 0.382812 0.150424 0.151095 2024-08-18T22:01:34.430 INFO:tasks.workunit.client.0.smithi099.stdout: 28 16 2978 2962 0.413166 0.4375 0.133643 0.150969 2024-08-18T22:01:35.430 INFO:tasks.workunit.client.0.smithi099.stdout: 29 16 3088 3072 0.413733 0.429688 0.109892 0.15093 2024-08-18T22:01:36.430 INFO:tasks.workunit.client.0.smithi099.stdout: 30 16 3202 3186 0.414783 0.445312 0.133938 0.150106 2024-08-18T22:01:37.430 INFO:tasks.workunit.client.0.smithi099.stdout:Total time run: 30.0831 2024-08-18T22:01:37.430 INFO:tasks.workunit.client.0.smithi099.stdout:Total writes made: 3202 2024-08-18T22:01:37.430 INFO:tasks.workunit.client.0.smithi099.stdout:Write size: 4096 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Object size: 4096 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Bandwidth (MB/sec): 0.415775 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Bandwidth: 0.0559845 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Max bandwidth (MB/sec): 0.5 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Min bandwidth (MB/sec): 0.257812 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Average IOPS: 106 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev IOPS: 14.332 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Max IOPS: 128 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Min IOPS: 66 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Average Latency(s): 0.150025 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Stddev Latency(s): 0.0437198 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Max latency(s): 0.610078 2024-08-18T22:01:37.431 INFO:tasks.workunit.client.0.smithi099.stdout:Min latency(s): 0.0990044 2024-08-18T22:01:37.481 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-08-18T22:01:42.485 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-08-18T22:01:42.485 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-08-18T22:01:42.669 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-08-18T22:01:42.669 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-08-18T22:01:42.669 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-08-18T22:01:42.670 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-08-18T22:01:42.670 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-08-18T22:01:42.670 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-08-18T22:01:42.670 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-08-18T22:01:42.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4281753600 2024-08-18T22:01:42.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 152088 2024-08-18T22:01:42.844 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-08-18T22:01:43.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 152088 2024-08-18T22:01:43.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (152088) - No such process 2024-08-18T22:01:43.845 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-08-18T22:01:44.309 INFO:tasks.workunit.client.0.smithi099.stderr:osd.0 is already down. 2024-08-18T22:01:44.325 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-08-18T22:01:44.359 INFO:tasks.workunit.client.0.smithi099.stdout:allocmap bluestore.allocmap 2024-08-18T22:01:46.960 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:01:46.957+0000 7f37af5b0ac0 -1 bluestore::NCB::operator()::(2)compare_allocators:: spillover 2024-08-18T22:01:46.960 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:01:46.957+0000 7f37af5b0ac0 -1 bluestore::NCB::compare_allocators::mismatch:: idx1=6 idx2=7 2024-08-18T22:01:46.960 INFO:tasks.workunit.client.0.smithi099.stderr:2024-08-18T22:01:46.957+0000 7f37af5b0ac0 -1 bluestore::NCB::read_allocation_from_drive_for_bluestore_tool::FAILURE. Allocator from file and allocator from metadata differ::ret=-1 2024-08-18T22:01:47.455 INFO:tasks.workunit.client.0.smithi099.stderr:allocmap failed: (1) Operation not permitted 2024-08-18T22:01:47.462 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: return 1 2024-08-18T22:01:47.463 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: return 1 2024-08-18T22:01:47.463 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2333: main: code=1 2024-08-18T22:01:47.463 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: teardown td/osd-bluefs-volume-ops 1 2024-08-18T22:01:47.463 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-08-18T22:01:47.463 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2024-08-18T22:01:47.463 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-08-18T22:01:47.464 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-08-18T22:01:47.464 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-08-18T22:01:47.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-08-18T22:01:47.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-08-18T22:01:47.464 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-08-18T22:01:47.572 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-08-18T22:01:47.573 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-08-18T22:01:47.573 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-08-18T22:01:47.574 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-08-18T22:01:47.575 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-08-18T22:01:47.575 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-08-18T22:01:47.575 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-08-18T22:01:47.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T22:01:47.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-08-18T22:01:47.576 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-08-18T22:01:47.576 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-18T22:01:47.577 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-08-18T22:01:47.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2024-08-18T22:01:47.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2024-08-18T22:01:47.578 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-08-18T22:01:47.579 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-bluefs-volume-ops/bluestore_tool.log td/osd-bluefs-volume-ops/mgr.x.log td/osd-bluefs-volume-ops/mon.a.log td/osd-bluefs-volume-ops/osd.0.log /home/ubuntu/cephtest/archive/log 2024-08-18T22:01:47.580 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-08-18T22:01:47.584 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-08-18T22:01:47.584 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-08-18T22:01:47.584 INFO:tasks.workunit.client.0.smithi099.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.97620 2024-08-18T22:01:47.584 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.97620 2024-08-18T22:01:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T22:01:47.586 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-08-18T22:01:47.586 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-08-18T22:01:47.586 INFO:tasks.workunit.client.0.smithi099.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2336: main: return 1 2024-08-18T22:01:47.586 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-08-18T22:01:47.586 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-08-18T22:01:47.910 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-08-18T22:01:48.087 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4bebd1fe22754086bb4643859d301588 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_964adda23da9f7b057cb34fdc0b82aeb7ed65324/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi099 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh' 2024-08-18T22:01:48.090 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-18T22:01:48.098 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-18T22:01:48.099 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-18T22:01:48.141 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-18T22:01:48.149 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-18T22:01:48.149 DEBUG:teuthology.orchestra.run.smithi099:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-18T22:01:48.174 INFO:teuthology.orchestra.run.smithi099.stdout: remote refid st t when poll reach delay offset jitter 2024-08-18T22:01:48.174 INFO:teuthology.orchestra.run.smithi099.stdout:============================================================================== 2024-08-18T22:01:48.174 INFO:teuthology.orchestra.run.smithi099.stdout:*hv01.front.sepi 67.205.162.81 3 u 34 64 377 0.079 -2.653 0.409 2024-08-18T22:01:48.174 INFO:teuthology.orchestra.run.smithi099.stdout:+hv02.front.sepi 63.231.80.2 3 u 20 64 377 0.082 -1.752 0.367 2024-08-18T22:01:48.174 INFO:teuthology.orchestra.run.smithi099.stdout:+hv03.front.sepi 74.6.168.72 3 u 35 64 377 0.088 -2.846 0.356 2024-08-18T22:01:48.175 INFO:teuthology.orchestra.run.smithi099.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-08-18T22:01:48.175 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-18T22:01:48.184 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-18T22:01:48.184 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-18T22:01:48.211 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-18T22:01:48.218 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-18T22:01:48.225 INFO:teuthology.task.internal:Duration was 3039.167256 seconds 2024-08-18T22:01:48.226 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-18T22:01:48.233 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-18T22:01:48.233 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-18T22:01:48.275 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-18T22:01:48.275 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2024-08-18T22:01:48.276 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-18T22:01:48.332 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-18T22:01:48.332 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-18T22:01:48.413 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-18T22:01:48.413 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-18T22:01:48.515 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-18T22:01:48.524 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-18T22:01:48.525 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-18T22:01:48.537 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-18T22:01:48.545 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-18T22:01:48.587 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2024-08-18T22:01:48.605 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2024-08-18T22:01:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T22:01:48.651 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-18T22:01:48.665 INFO:teuthology.task.internal:Transferring archived files... 2024-08-18T22:01:48.667 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-08-18_20:00:17-rados-main-distro-default-smithi/7859229/remote/smithi099 2024-08-18T22:01:48.668 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-18T22:01:51.024 INFO:teuthology.task.internal:Removing archive directory... 2024-08-18T22:01:51.024 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf -- /home/ubuntu/cephtest/archive 2024-08-18T22:01:51.082 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-18T22:01:51.091 INFO:teuthology.task.internal:Not uploading archives. 2024-08-18T22:01:51.092 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-18T22:01:51.171 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-18T22:01:51.172 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-18T22:01:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: 658616 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 18 22:01 /home/ubuntu/cephtest 2024-08-18T22:01:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: 675048 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 18 21:20 /home/ubuntu/cephtest/mnt.0 2024-08-18T22:01:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: 677074 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 18 21:48 /home/ubuntu/cephtest/mnt.0/client.0 2024-08-18T22:01:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: 792938 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Aug 18 21:48 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-08-18T22:01:51.179 INFO:teuthology.orchestra.run.smithi099.stdout: 792939 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Aug 18 22:01 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-08-18T22:01:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-18T22:01:51.180 INFO:teuthology.orchestra.run.smithi099.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-08-18T22:01:51.181 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_67db276914d29505c9707d04f3a3b8fb9d8aee4b/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi099 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-18T22:01:51.185 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-18T22:01:51.195 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-18T22:01:51.269 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi099 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-18T22:01:51.272 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} duration: 3039.16725564003 failure_reason: 'Command failed (workunit test osd/osd-bluefs-volume-ops.sh) on smithi099 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=964adda23da9f7b057cb34fdc0b82aeb7ed65324 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4bebd1fe22754086bb4643859d301588 status: fail success: false 2024-08-18T22:01:51.273 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-18T22:01:51.353 INFO:teuthology.run:FAIL